From igorsyl at gmail.com  Thu Jan  1 14:09:30 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Thu, 1 Jan 2009 13:09:30 -0600
Subject: [Numpy-discussion] PyArray_DescrConverter segfault
Message-ID: <b3c85b560901011109va950f5k689da9c87cbcf20@mail.gmail.com>

Hi,
Does anyone have an insight on the following problem?

PyObject* descr(PyObject* self, PyObject* args) {
  PyObject *d;
  PyArg_ParseTuple(args, "O&", PyArray_DescrConverter, &d);
  return d;
}

>>> import numpy
>>> import mymodule
>>> numpy.__version__
'1.2.1'
>>> mymodule.descr([('a', 'i4'), ('b', 'i8')])
segmentation fault (core dumped)  ipython

Thanks!
-Igor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090101/240bda46/attachment.html>

From igorsyl at gmail.com  Thu Jan  1 14:21:20 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Thu, 1 Jan 2009 13:21:20 -0600
Subject: [Numpy-discussion] Fwd: PyArray_DescrConverter segfault
In-Reply-To: <b3c85b560901011109va950f5k689da9c87cbcf20@mail.gmail.com>
References: <b3c85b560901011109va950f5k689da9c87cbcf20@mail.gmail.com>
Message-ID: <b3c85b560901011121h347d9f9enb2a91e53ca27ea8c@mail.gmail.com>

I realize I need to call import_array() in the module initialization
module.  Why isn't this equivalent to importing numpy before importing my
module?

---------- Forwarded message ----------
From: Igor Sylvester <igorsyl at gmail.com>
Date: Thu, Jan 1, 2009 at 1:09 PM
Subject: PyArray_DescrConverter segfault
To: Numpy-discussion at scipy.org


Hi,
Does anyone have an insight on the following problem?

PyObject* descr(PyObject* self, PyObject* args) {
  PyObject *d;
  PyArg_ParseTuple(args, "O&", PyArray_DescrConverter, &d);
  return d;
}

>>> import numpy
>>> import mymodule
>>> numpy.__version__
'1.2.1'
>>> mymodule.descr([('a', 'i4'), ('b', 'i8')])
segmentation fault (core dumped)  ipython

Thanks!
-Igor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090101/348fcf37/attachment.html>

From robert.kern at gmail.com  Thu Jan  1 17:11:10 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 1 Jan 2009 17:11:10 -0500
Subject: [Numpy-discussion] Fwd: PyArray_DescrConverter segfault
In-Reply-To: <b3c85b560901011121h347d9f9enb2a91e53ca27ea8c@mail.gmail.com>
References: <b3c85b560901011109va950f5k689da9c87cbcf20@mail.gmail.com>
	<b3c85b560901011121h347d9f9enb2a91e53ca27ea8c@mail.gmail.com>
Message-ID: <3d375d730901011411w21b534e5mc5f35ca39f8e0ff7@mail.gmail.com>

On Thu, Jan 1, 2009 at 14:21, Igor Sylvester <igorsyl at gmail.com> wrote:
> I realize I need to call import_array() in the module initialization
> module.  Why isn't this equivalent to importing numpy before importing my
> module?

In 3rd party extension modules, all of the PyArray_* API functions are
actually #define macros pointing to an array of function pointers.
import_array() imports numpy.core.multiarray and sets up this array.
If you don't do this, then trying to call one of the PyArray_*
functions will result in a segfault because it tries to dereference a
pointer in the array that is not set up.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From igorsyl at gmail.com  Thu Jan  1 17:25:21 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Thu, 1 Jan 2009 16:25:21 -0600
Subject: [Numpy-discussion] Fwd: PyArray_DescrConverter segfault
In-Reply-To: <3d375d730901011411w21b534e5mc5f35ca39f8e0ff7@mail.gmail.com>
References: <b3c85b560901011109va950f5k689da9c87cbcf20@mail.gmail.com>
	<b3c85b560901011121h347d9f9enb2a91e53ca27ea8c@mail.gmail.com>
	<3d375d730901011411w21b534e5mc5f35ca39f8e0ff7@mail.gmail.com>
Message-ID: <b3c85b560901011425uc3edf56nded5217e2ed56221@mail.gmail.com>

Robert,

If I import numpy and then import a 3rd party extension module, why does the
extension module still have to call import_array if numpy sets this array in
the first place?  I assume that there's a single array of API functions in a
single python process with multiple extension modules.

Igor.

On Thu, Jan 1, 2009 at 4:11 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Thu, Jan 1, 2009 at 14:21, Igor Sylvester <igorsyl at gmail.com> wrote:
> > I realize I need to call import_array() in the module initialization
> > module.  Why isn't this equivalent to importing numpy before importing my
> > module?
>
> In 3rd party extension modules, all of the PyArray_* API functions are
> actually #define macros pointing to an array of function pointers.
> import_array() imports numpy.core.multiarray and sets up this array.
> If you don't do this, then trying to call one of the PyArray_*
> functions will result in a segfault because it tries to dereference a
> pointer in the array that is not set up.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090101/e491a51f/attachment.html>

From robert.kern at gmail.com  Thu Jan  1 17:27:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 1 Jan 2009 17:27:40 -0500
Subject: [Numpy-discussion] Fwd: PyArray_DescrConverter segfault
In-Reply-To: <b3c85b560901011425uc3edf56nded5217e2ed56221@mail.gmail.com>
References: <b3c85b560901011109va950f5k689da9c87cbcf20@mail.gmail.com>
	<b3c85b560901011121h347d9f9enb2a91e53ca27ea8c@mail.gmail.com>
	<3d375d730901011411w21b534e5mc5f35ca39f8e0ff7@mail.gmail.com>
	<b3c85b560901011425uc3edf56nded5217e2ed56221@mail.gmail.com>
Message-ID: <3d375d730901011427n6b09f8ect1ea44c31f65785f3@mail.gmail.com>

On Thu, Jan 1, 2009 at 17:25, Igor Sylvester <igorsyl at gmail.com> wrote:
> Robert,
>
> If I import numpy and then import a 3rd party extension module, why does the
> extension module still have to call import_array if numpy sets this array in
> the first place?  I assume that there's a single array of API functions in a
> single python process with multiple extension modules.

Your 3rd party extension module does not know the location of that
array until you call import_array().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From mlandis001 at comcast.net  Fri Jan  2 13:29:26 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Fri, 02 Jan 2009 13:29:26 -0500
Subject: [Numpy-discussion] building numpy/scipy
Message-ID: <20090102183005.27311C7C019@scipy.org>

Some of the install instructions are kind of ambiguous.

When a library name ends in .a or .dll, it's obvious what it is, but 
'library' is sometimes used generically without indicating whether 
you're talking about static or dynamic, e.g. how does numpy/scipy 
link to MKL?  Is it statically or dynamically linked?

It's not clear where things should go or how things are found.  After 
downloading numpy and putting numpy-1.2.1 into a directory named 
numpy-1.2.1 in python2.5/Lib/site-packages/ and running python 
setup.py install from the numpy-1.2.1 directory, you get a numpy 
directory in site-packages.  Later, the install instructions refer to 
the numpy directory generically.  Which numpy directory are you 
talking about?  Should the numpy-1.2.1 directory not be in 
site-packages in the first place?  Ditto for scipy.  E.g. are you 
referring to the site.cfg in the distribution directory (numpy-1.2.1) 
or the installed directory (numpy without version notation).  Maybe 
you should give names to these directories (e.g. distribution and 
install) and always refer to them with leading adjectives.

It should be possible to create a script that keeps track of all of 
these extra lib configuration steps, so installs won't be so involved 
and error prone.

I built ATLAS (they need better documentation of processor types and 
how to use what CPU-Z tells you) and I have MKL installed.  A quarter 
of my MKL evaluation days have already ticked away, and I've come to 
the disappointing realization that 30 days is probably not going to 
be enough to get the scipy/MKL install right.

What's compatible with what?  There is mention of MKL, ATLAS, LAPACK 
and CLAPACK (I also downloaded CLAPACK 3.1.1.1).  It seems like 
CLAPACK would be naturally compatible because python uses C-style 
matrix addressing, but does Python (or numpy) flip indexes around if 
you have LAPACK instead of CLAPACK, so it works anyway, if slower?

There is mention of ".numpy-site.cfg" in the user's home directory 
(which I created), but it didn't seem to have any effect on installing.

I created the following site.cfg in the numpy (install) directory:

[mkl]
include_dirs = D:\Programs\Intel\MKL\10.1.0.018\include
library_dirs = D:\Programs\Intel\MKL\10.1.0.018\ia32\lib
mkl_libs = mkl_ia32, mkl_c_dll, libguide40
lapack_libs = mkl_lapack

Do numpy AND scipy each need a site.cfg in their respective install 
directories?  If so, does one take precedence over the other?

When I run 'python setup.py install' from the numpy install directory, I get:

Warning: No configuration returned, assuming unavailable.
blas_opt_info:
blas_mkl_info:
   libraries mkl, vml, guide not found in d:\Programs\Python25\lib
   libraries mkl, vml, guide not found in C:\
   libraries mkl, vml, guide not found in d:\Programs\Python25\libs
   NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
   libraries mkl, vml, guide not found in d:\Programs\Python25\lib
   libraries mkl, vml, guide not found in C:\
   libraries mkl, vml, guide not found in d:\Programs\Python25\libs
   NOT AVAILABLE

atlas_blas_info:
   libraries mkl, vml, guide not found in d:\Programs\Python25\lib
   libraries mkl, vml, guide not found in C:\
   libraries mkl, vml, guide not found in d:\Programs\Python25\libs
   NOT AVAILABLE

D:\Programs\Python25\lib\site-packages\numpy\disutils\system_info.py:1340: 
UserWarning:
     Atlas (http://math-atlas.sourceforgr.net/) libraries not found
     Directories to search for the libraries can be specified in the 
numpy/disutils/site.cfg file (section [atlas]) or by setting the 
ATLAS environment variable.
   warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
   libraries blas not found in d:\Programs\Python25\lib
-----------------------------------------------------------------
Where is it looking for blas_opt_info, blas_mkl_info, 
atlas_blas_threads_info, and atlas_blas_info?  Is it actually looking 
on sourceforge for Atlas?  FYI, when I try using make in 
...MKL/10.1.0.018/examples/cblas, I get a "missing separator" error 
on the "!INCLUDE cblas.lst" line (first significant line in the 
file).  There are no spaces on this line.  Could cygwin make be 
complaining about something inside the cblas.lst file, or is 
complaining about the line not being terminated the way it expects it 
to be (I see carriage return, line feed pairs in the file).

HELP!!!!



From cournape at gmail.com  Fri Jan  2 13:43:55 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 3 Jan 2009 03:43:55 +0900
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090102183005.27311C7C019@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
Message-ID: <5b8d13220901021043x34b5f823r58aa09c9061c925@mail.gmail.com>

On Sat, Jan 3, 2009 at 3:29 AM, Mike Landis <mlandis001 at comcast.net> wrote:
> Some of the install instructions are kind of ambiguous.
>
> When a library name ends in .a or .dll, it's obvious what it is, but
> 'library' is sometimes used generically without indicating whether
> you're talking about static or dynamic, e.g. how does numpy/scipy
> link to MKL?  Is it statically or dynamically linked?

Dynamically first, statically if only the static library is found.

>
> It's not clear where things should go or how things are found.  After
> downloading numpy and putting numpy-1.2.1 into a directory named
> numpy-1.2.1 in python2.5/Lib/site-packages/ and running python
> setup.py install from the numpy-1.2.1 directory, you get a numpy
> directory in site-packages.  Later, the install instructions refer to
> the numpy directory generically.  Which numpy directory are you
> talking about?  Should the numpy-1.2.1 directory not be in
> site-packages in the first place?

No, it should certainly not be in site-packages. Why would you want to
do that ? site-packages is reserved for already built python modules.
It is a very bad idea to put anything else there.

>
> It should be possible to create a script that keeps track of all of
> these extra lib configuration steps, so installs won't be so involved
> and error prone.

There is no ambiguity: when referring to installation, it is
understood we refer to the source directory, always.

>
> What's compatible with what?  There is mention of MKL, ATLAS, LAPACK
> and CLAPACK (I also downloaded CLAPACK 3.1.1.1).  It seems like
> CLAPACK would be naturally compatible because python uses C-style
> matrix addressing, but does Python (or numpy) flip indexes around if
> you have LAPACK instead of CLAPACK, so it works anyway, if slower?

Numpy and Scipy supports the netlib F77 interface to both blas and
lapack. There is no real advantage to using CLAPACK/CBLAS instead of
those if you have the fortran interfaces.

>
> There is mention of ".numpy-site.cfg" in the user's home directory
> (which I created), but it didn't seem to have any effect on installing.
>
> I created the following site.cfg in the numpy (install) directory:
>
> [mkl]
> include_dirs = D:\Programs\Intel\MKL\10.1.0.018\include
> library_dirs = D:\Programs\Intel\MKL\10.1.0.018\ia32\lib
> mkl_libs = mkl_ia32, mkl_c_dll, libguide40
> lapack_libs = mkl_lapack

There are many problems with the MLK - and I bet it does not work on
windows. You are much better of using the numpy and scipy binaries.

>
> Do numpy AND scipy each need a site.cfg in their respective install
> directories?  If so, does one take precedence over the other?

When building, the site.cfg should be put in the *source* directory.

But again, particularly on windows, you should really use the
distributed binaries. Building numpy and scipy with external
BLAS/LAPACK is not easy, especially on windows.

David


From matthieu.brucher at gmail.com  Fri Jan  2 13:45:29 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 2 Jan 2009 19:45:29 +0100
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090102183005.27311C7C019@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
Message-ID: <e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>

2009/1/2 Mike Landis <mlandis001 at comcast.net>:
> Some of the install instructions are kind of ambiguous.
>
> When a library name ends in .a or .dll, it's obvious what it is, but
> 'library' is sometimes used generically without indicating whether
> you're talking about static or dynamic, e.g. how does numpy/scipy
> link to MKL?  Is it statically or dynamically linked?

If the dynamic libraries are available, they are used.

> It's not clear where things should go or how things are found.  After
> downloading numpy and putting numpy-1.2.1 into a directory named
> numpy-1.2.1 in python2.5/Lib/site-packages/ and running python
> setup.py install from the numpy-1.2.1 directory, you get a numpy
> directory in site-packages.

Wait, never put numpy-1.2.1 inside site-packes, build it somewhere else.

  Later, the install instructions refer to
> the numpy directory generically.  Which numpy directory are you
> talking about?  Should the numpy-1.2.1 directory not be in
> site-packages in the first place?  Ditto for scipy.  E.g. are you
> referring to the site.cfg in the distribution directory (numpy-1.2.1)
> or the installed directory (numpy without version notation).

Just get inside numpy-1.2.1, type python setup.py install and you
never have to use this folder. The only relevant folder is the numpy
one inside site-packages.

 Maybe
> you should give names to these directories (e.g. distribution and
> install) and always refer to them with leading adjectives.

Perhaps.

> It should be possible to create a script that keeps track of all of
> these extra lib configuration steps, so installs won't be so involved
> and error prone.

I just use python setup.py install, nothing else, and it works with
the default Linux mathematic libraries.

> I built ATLAS (they need better documentation of processor types and
> how to use what CPU-Z tells you) and I have MKL installed.  A quarter
> of my MKL evaluation days have already ticked away, and I've come to
> the disappointing realization that 30 days is probably not going to
> be enough to get the scipy/MKL install right.

I don't know if many people are using Numpy with the MKL on Windows.

> What's compatible with what?  There is mention of MKL, ATLAS, LAPACK
> and CLAPACK (I also downloaded CLAPACK 3.1.1.1).  It seems like
> CLAPACK would be naturally compatible because python uses C-style
> matrix addressing, but does Python (or numpy) flip indexes around if
> you have LAPACK instead of CLAPACK, so it works anyway, if slower?

CLAPACK is not a standard interface, so it is not usable with MKL or ACML.

> There is mention of ".numpy-site.cfg" in the user's home directory
> (which I created), but it didn't seem to have any effect on installing.

Only site.cfg is relevant.

> I created the following site.cfg in the numpy (install) directory:
>
> [mkl]
> include_dirs = D:\Programs\Intel\MKL\10.1.0.018\include
> library_dirs = D:\Programs\Intel\MKL\10.1.0.018\ia32\lib
> mkl_libs = mkl_ia32, mkl_c_dll, libguide40
> lapack_libs = mkl_lapack



> Do numpy AND scipy each need a site.cfg in their respective install
> directories?  If so, does one take precedence over the other?

Yes and I don't know ;)

> When I run 'python setup.py install' from the numpy install directory, I get:
>
> Warning: No configuration returned, assuming unavailable.
> blas_opt_info:
> blas_mkl_info:
>   libraries mkl, vml, guide not found in d:\Programs\Python25\lib
>   libraries mkl, vml, guide not found in C:\
>   libraries mkl, vml, guide not found in d:\Programs\Python25\libs
>   NOT AVAILABLE
>
> atlas_blas_threads_info:
> Setting PTATLAS=ATLAS
>   libraries mkl, vml, guide not found in d:\Programs\Python25\lib
>   libraries mkl, vml, guide not found in C:\
>   libraries mkl, vml, guide not found in d:\Programs\Python25\libs
>   NOT AVAILABLE
>
> atlas_blas_info:
>   libraries mkl, vml, guide not found in d:\Programs\Python25\lib
>   libraries mkl, vml, guide not found in C:\
>   libraries mkl, vml, guide not found in d:\Programs\Python25\libs
>   NOT AVAILABLE
>
> D:\Programs\Python25\lib\site-packages\numpy\disutils\system_info.py:1340:
> UserWarning:
>     Atlas (http://math-atlas.sourceforgr.net/) libraries not found
>     Directories to search for the libraries can be specified in the
> numpy/disutils/site.cfg file (section [atlas]) or by setting the
> ATLAS environment variable.
>   warnings.warn(AtlasNotFoundError.__doc__)
> blas_info:
>   libraries blas not found in d:\Programs\Python25\lib
> -----------------------------------------------------------------
> Where is it looking for blas_opt_info, blas_mkl_info,
> atlas_blas_threads_info, and atlas_blas_info?  Is it actually looking
> on sourceforge for Atlas?

it looks in site.cfg for those informations.

  FYI, when I try using make in
> ...MKL/10.1.0.018/examples/cblas, I get a "missing separator" error
> on the "!INCLUDE cblas.lst" line (first significant line in the
> file).  There are no spaces on this line.  Could cygwin make be
> complaining about something inside the cblas.lst file, or is
> complaining about the line not being terminated the way it expects it
> to be (I see carriage return, line feed pairs in the file).

Did you try mingw or Visual studio 2003?

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From boogaloojb at yahoo.fr  Fri Jan  2 13:47:11 2009
From: boogaloojb at yahoo.fr (Jean-Baptiste Rudant)
Date: Fri, 2 Jan 2009 18:47:11 +0000 (GMT)
Subject: [Numpy-discussion] Re :  Alternative to record array
References: <602202.94406.qm@web28502.mail.ukl.yahoo.com>
	<200812300000.10365.faltet@pytables.org>
	<200812301634.28218.faltet@pytables.org>
Message-ID: <395702.84148.qm@web28512.mail.ukl.yahoo.com>

Thank you for everything, it works fine ant it is very helpful.

Regards,

Jean-Baptiste Rudant



________________________________
De : Francesc Alted <faltet at pytables.org>
? : Discussion of Numerical Python <numpy-discussion at scipy.org>
Envoy? le : Mardi, 30 D?cembre 2008, 16h34mn 27s
Objet : Re: [Numpy-discussion] Alternative to record array

A Tuesday 30 December 2008, Francesc Alted escrigu?:
> A Monday 29 December 2008, Jean-Baptiste Rudant escrigu?:
[snip]
>
> The difference for both approaches is that the row-wise arrangement
> is more efficient when data is iterated by field, while the
> column-wise one is more efficient when data is iterated by column. 
> This is why you are seeing the increase of 4x in performance
> --incidentally, by looking at both data arrangements, I'd expect an
> increase of just 2x (the stride count is 2 in this case), but I
> suspect that there are hidden copies during the increment operation
> for the record array case.

As I was mystified about this difference in speed, I kept investigating 
and I think I have an answer for the difference in the expected 
speed-up in the unary increment operator over a recarray field.  After 
looking at the numpy code, it turns out that the next statement:

data.ages += 1

is more or less equivalent to:

a = data.ages
a[:] = a + 1

i.e. a temporary is created (for keeping the result of 'a + 1') and then 
assigned to the 'ages' column.  As it happens that, in this sort of 
operations, the memory copies are the bottleneck, the creation of the 
first temporary introduced a slowdown of 2x (due to the strided column) 
and the assignment represents the additional 2x (4x in total).  
However, the next idiom:

a = data.ages
a += 1

effectively removes the need for the temporary copy and is 2x faster 
than the original "data.ages += 1".  This can be seen in the next 
simple benchmark:

---------------------------
import numpy, timeit

count = 10e6
ages  = numpy.random.randint(0,100,count)
weights = numpy.random.randint(1,200,count)
data = numpy.rec.fromarrays((ages,weights),names='ages,weights')

timer = timeit.Timer('data.ages += 1','from __main__ import data')
print "v0-->", timer.timeit(number=10)
timer = timeit.Timer('a=data.ages; a[:] = a + 1','from __main__ import 
data')
print "v1-->", timer.timeit(number=10)
timer = timeit.Timer('a=data.ages; a += 1','from __main__ import data')
print "v2-->", timer.timeit(number=10)
timer = timeit.Timer('ages += 1','from __main__ import ages')
print "v3-->", timer.timeit(number=10)
---------------------------

which produces the next output on my laptop:

v0--> 2.98340201378
v1--> 3.22748112679
v2--> 1.5474319458
v3--> 0.809724807739

As a final comment, I suppose that unary operators (+=, -=...) can be 
optimized in the context of recarray columns in numpy, but I don't 
think it is worth the effort:  when really high performance is needed 
for operating with columns in the context of recarrays, a column-wise 
approach is best.

Cheers,

-- 
Francesc Alted
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090102/18b73341/attachment.html>

From mlandis001 at comcast.net  Fri Jan  2 18:27:46 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Fri, 02 Jan 2009 18:27:46 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <5b8d13220901021043x34b5f823r58aa09c9061c925@mail.gmail.com
 >
References: <20090102183005.27311C7C019@scipy.org>
	<5b8d13220901021043x34b5f823r58aa09c9061c925@mail.gmail.com>
Message-ID: <20090102232825.F26BBC8410D@scipy.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090102/1454d7ff/attachment.html>

From mlandis001 at comcast.net  Fri Jan  2 20:45:19 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Fri, 02 Jan 2009 20:45:19 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090102232825.F26BBC8410D@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<5b8d13220901021043x34b5f823r58aa09c9061c925@mail.gmail.com>
	<20090102232825.F26BBC8410D@scipy.org>
Message-ID: <20090103014559.81970C7C009@scipy.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090102/c6ed723f/attachment.html>

From mlandis001 at comcast.net  Fri Jan  2 22:10:07 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Fri, 02 Jan 2009 22:10:07 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com
 >
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
Message-ID: <20090103031048.38829C7C009@scipy.org>

Maybe the reason I'm having trouble is that I'm trying to get it 
working on Windows, when almost everyone else is running on Linux?  I 
have cygwin with f77, g++, make, ... installed, but it's definitely 
not a Linux machine.  I'm working from the windows install 
documentation page.  Maybe there are additional steps that you 
wouldn't have on Linux.  It's not clear whether BLAS and LAPACK and 
ATLAS are all distinct or if building one gets you one or more of the 
others, e.g. the ATLAS build produces both blas and lapack 
directories.  Is there a third ATLAS library?

When I set up site.cfg in site-packages/numpy and run "python 
setup.py install" from there, it complains that this isn't the right 
place to run setup.py, so I put the site.cfg in d:\temp\numpy-1.2.1\ 
and ran setup.py install there.  I now have ATLAS libraries, so my 
site.cfg looks like this:

[atlas]
library_dirs = d:\Docs\ATLAS\build\lib
atlas_libs = lapack, f77blas, cblas, atlas
-----------------------------------------------------
Some good news, some bad, details below.  Any suggestions would be 
appreciated...

Thanks, Mike

Running from numpy source directory.
blas_opt_info:
blas_mkl_info:
    libraries mkl, vml, guide not found in d:\Programs\Python25\lib
    libraries mkl, vml, guide not found in C:\
    libraries mkl, vml, guide not found in d:\Programs\Python25\libs
    NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
   FOUND:
      libraries = ['lapack', 'f77blas', ;cblas', 'atlas']
      library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
      language = c

No module named msvccompiler in numpy.disutils; trying from disutils
   FOUND:
      libraries = ['lapack', 'f77blas', ;cblas', 'atlas']
      library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
      language = c
      define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]

lapack_opt_info:
lapack_mkl_info:
mkl_info:
    libraries mkl, vml, guide not found in d:\Programs\Python25\lib
    libraries mkl, vml, guide not found in C:\
    libraries mkl, vml, guide not found in d:\Programs\Python25\libs
    NOT AVAILABLE

    NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
   libraries lapack_atlas not found in d:\Docs\ATLAS\build\lib
numpy.disutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
d:\temp\numpy-1.2.1\numpy\disutils\system_info.py:955: UserWarning:
********************************************************************************
      Lapack library (from ATLAS) is probably incomplete:
          size of d:\Docs\ATLAS\build\lib\libapack.a is 251k (expected >4000k)

      Follow the instructions in the KNOWN PROBLEMS section of the file
      numpy/INSTALL.txt.  [  NOTE:  There is no such section in this file.  ]
********************************************************************************

   warnings.warn(message)
Setting PTATLAS=ATLAS
   FOUND:
      libraries = ['lapack', 'lapack', 'f77blas', ;cblas', 'atlas']
      library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
      language = c

No module named msvccompiler in numpy.disutils; trying from disutils
   FOUND:
      libraries = ['lapack', 'lapack', 'f77blas', ;cblas', 'atlas']
      library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
      language = c
      define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]

could not resolve pattern in '': '*.txt'
non-existing path in '': 'COMPATIBILITY'
running install
running build
running config_cc
unifying config_cc, config, build_clib, build_ext, build commands 
--compiler options
running config_fc
unifying config_fc, config, build_clib, build_ext, build commands 
--fcompiler options
running build_src
building py_modules sources
building extension "numpy.core.multiarray" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_numpy_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__multiarray_api.h' to sources.
   adding 'build\src.win32-2.5\numpy\core\src' to include_dirs.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\src\\scalartypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\src\\arraytypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__multiarray_api.h']
building extension "numpy.core.umath" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_ufunc_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__ufunc_api.h' to sources.
   adding 'build\src.win32-2.5\numpy\core\src' to include_dirs.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\src\\scalartypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\src\\arraytypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__ufunc_api.h']
building extension "numpy.core._sort" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_numpy_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__multiarray_api.h' to sources.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__multiarray_api.h']
building extension "numpy.core.scalarmath" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_numpy_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__multiarray_api.h' to sources.
executing numpy\core\code_generators\generate_ufunc_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__ufunc_api.h' to sources.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__multiarray_api.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__ufunc_api.h']
building extension "numpy.core._dotblas" sources
   adding 'numpy\core\blasdot\_dotblas.c' to sources.
building extension "numpy.lib._compiled_base" sources
building extension "numpy.numarray._capi" sources
building extension "numpy.fft.fftpack_lite" sources
building extension "numpy.linalg.lapack_lite" sources
### Warning:  python_xerbla.c is disabled ###
   adding 'numpy\linalg\lapack_litemodule.c' to sources.
building extension "numpy.random.mtrand" sources
Traceback (most recent call last):
   File "setup.py", line 96, in <module>
     setup_package()
   File "setup.py", line 89, in setup_package
     configuration=configuration )
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\core.py", 
line 184, in setup
   File "d:\programs\python25\lib\distutils\core.py", line 151, in setup
     dist.run_commands()
   File "d:\programs\python25\lib\distutils\dist.py", line 974, in run_commands
     self.run_command(cmd)
   File "d:\programs\python25\lib\distutils\dist.py", line 994, in run_command
     cmd_obj.run()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\install.py", 
line 49, in run
   File "d:\programs\python25\lib\distutils\command\install.py", line 
506, in run
     self.run_command('build')
   File "d:\programs\python25\lib\distutils\cmd.py", line 333, in run_command
     self.distribution.run_command(command)
   File "d:\programs\python25\lib\distutils\dist.py", line 994, in run_command
     cmd_obj.run()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build.py", 
line 37, in run
   File "d:\programs\python25\lib\distutils\command\build.py", line 112, in run
     self.run_command(cmd_name)
   File "d:\programs\python25\lib\distutils\cmd.py", line 333, in run_command
     self.distribution.run_command(command)
   File "d:\programs\python25\lib\distutils\dist.py", line 994, in run_command
     cmd_obj.run()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 130, in run
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 147, in build_sources
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 250, in build_extension_sources
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 307, in generate_sources
   File "numpy\random\setup.py", line 11, in generate_libraries
     if config_cmd.try_run(tc):
   File "d:\programs\python25\lib\distutils\command\config.py", line 
278, in try_run
     self._check_compiler()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\config.py", 
line 26, in _check_compiler
   File "d:\programs\python25\lib\distutils\command\config.py", line 
107, in _check_compiler
     dry_run=self.dry_run, force=1)
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\ccompiler.py", 
line 366, in new_compiler
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\mingw32ccompiler.py", 
line 46, in __init__
   File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 
84, in __init__
     get_versions()
   File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 
424, in get_versions
     ld_version = StrictVersion(result.group(1))
   File "d:\programs\python25\lib\distutils\version.py", line 40, in __init__
     self.parse(vstring)
   File "d:\programs\python25\lib\distutils\version.py", line 107, in parse
     raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '2.18.50.20080625'



From tgrav at mac.com  Fri Jan  2 23:05:15 2009
From: tgrav at mac.com (Tommy Grav)
Date: Fri, 02 Jan 2009 23:05:15 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103031048.38829C7C009@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
Message-ID: <524D03AE-2D0E-4A9A-8FE7-BBB700BC34D9@mac.com>

>

Is there any reason why you can not use the numpy-1.2.1-win32- 
superpack-python2.4.exe
from the http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
download page? I think that is what Mr. Kern meant by using the  
binaries. This will install
already built code into the proper places on your Windows box.

Cheers
   Tommy


From mlandis001 at comcast.net  Fri Jan  2 23:26:52 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Fri, 02 Jan 2009 23:26:52 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <524D03AE-2D0E-4A9A-8FE7-BBB700BC34D9@mac.com>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<524D03AE-2D0E-4A9A-8FE7-BBB700BC34D9@mac.com>
Message-ID: <20090103042733.2F910C7C009@scipy.org>

Have to use Pyton 2.5 because I'm also using web2py.  Python 2.5 and 
a bunch of packages that depend on it are already installed.

At 11:05 PM 1/2/2009, you wrote:
> >
>
>Is there any reason why you can not use the numpy-1.2.1-win32-
>superpack-python2.4.exe
>from the 
>http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
>download page? I think that is what Mr. Kern meant by using the
>binaries. This will install
>already built code into the proper places on your Windows box.
>
>Cheers
>    Tommy
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at scipy.org
>http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>No virus found in this incoming message.
>Checked by AVG - http://www.avg.com
>Version: 8.0.176 / Virus Database: 270.10.2/1871 - Release Date: 
>1/1/2009 5:01 PM



From tgrav at mac.com  Fri Jan  2 23:45:01 2009
From: tgrav at mac.com (Tommy Grav)
Date: Fri, 02 Jan 2009 23:45:01 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103042733.2F910C7C009@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<524D03AE-2D0E-4A9A-8FE7-BBB700BC34D9@mac.com>
	<20090103042733.2F910C7C009@scipy.org>
Message-ID: <81A59E45-69B7-4351-B44C-95441A5A9F83@mac.com>

There is a superpack for the python2.5 at the same page. Again a  
binary .exe
file that should make the installing a fair bit easier.

Cheers
    Tommy


On Jan 2, 2009, at 11:26 PM, Mike Landis wrote:

> Have to use Pyton 2.5 because I'm also using web2py.  Python 2.5 and
> a bunch of packages that depend on it are already installed.
>
> At 11:05 PM 1/2/2009, you wrote:
>>>
>>
>> Is there any reason why you can not use the numpy-1.2.1-win32-
>> superpack-python2.4.exe
>> from the
>> http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
>> download page? I think that is what Mr. Kern meant by using the
>> binaries. This will install
>> already built code into the proper places on your Windows box.
>>
>> Cheers
>>   Tommy
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>> No virus found in this incoming message.
>> Checked by AVG - http://www.avg.com
>> Version: 8.0.176 / Virus Database: 270.10.2/1871 - Release Date:
>> 1/1/2009 5:01 PM
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From mlandis001 at comcast.net  Sat Jan  3 00:07:19 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 00:07:19 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <81A59E45-69B7-4351-B44C-95441A5A9F83@mac.com>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<524D03AE-2D0E-4A9A-8FE7-BBB700BC34D9@mac.com>
	<20090103042733.2F910C7C009@scipy.org>
	<81A59E45-69B7-4351-B44C-95441A5A9F83@mac.com>
Message-ID: <20090103050805.49D18C7C009@scipy.org>

Does it sort of non-destructively overlay the 2.5 that I'm already running?

At 11:45 PM 1/2/2009, you wrote:
>There is a superpack for the python2.5 at the same page. Again a 
>binary .exe file that should make the installing a fair bit easier.
>
>Cheers
>     Tommy
>
>On Jan 2, 2009, at 11:26 PM, Mike Landis wrote:
>
> > Have to use Pyton 2.5 because I'm also using web2py.  Python 2.5 and
> > a bunch of packages that depend on it are already installed.
> >
> > At 11:05 PM 1/2/2009, you wrote:
> >> Is there any reason why you can not use the numpy-1.2.1-win32-
> >> superpack-python2.4.exe
> >> from the
> >> 
> http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
> >> download page? I think that is what Mr. Kern meant by using the
> >> binaries. This will install
> >> already built code into the proper places on your Windows box.
> >>
> >> Cheers
> >>   Tommy
> >> _______________________________________________
> >> Numpy-discussion mailing list
> >> Numpy-discussion at scipy.org
> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >> No virus found in this incoming message.
> >> Checked by AVG - http://www.avg.com
> >> Version: 8.0.176 / Virus Database: 270.10.2/1871 - Release Date:
> >> 1/1/2009 5:01 PM
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at scipy.org
>http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>No virus found in this incoming message.
>Checked by AVG - http://www.avg.com
>Version: 8.0.176 / Virus Database: 270.10.2/1871 - Release Date: 
>1/1/2009 5:01 PM



From mlandis001 at comcast.net  Sat Jan  3 00:11:11 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 00:11:11 -0500
Subject: [Numpy-discussion] installing numpy with Python2.5 already present
Message-ID: <20090103051153.50F33C7C009@scipy.org>

This time I included both ATLAS and MKL in the site.cfg file and got 
a little further...  D:\temp\numpy-1.2.1\site.cfg now looks like:

[atlas]
library_dirs = d:\Docs\ATLAS\build\lib
atlas_libs = lapack, f77blas, cblas, atlas

[mkl]
include_dirs = D:\Programs\Intel\MKL\10.1.0.018\include
library_dirs = D:\Programs\Intel\MKL\10.1.0.018\ia32\lib
mkl_libs = mkl_ia32, mkl_c_dll, libguide40
lapack_libs = mkl_lapack
-----------------------------------------------------------------------------------
FOUND a few more things...  though still having problems finding 
blas_opt_info, blas_mkl_info (even though it finds MKL 
later???)  Install log follows...
-----------------------------------------------------------------------------------
Running from numpy source directory.
F2PY Version 2_5972
blas_opt_info:
blas_mkl_info:
   libraries mkl_ia32,mkl_c_dll,libguide40 not found in 
D:\Programs\Intel\MKL\10.1.0.018\ia32\lib
   NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
   FOUND:
     libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
     library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
     language = c

No module named msvccompiler in numpy.distutils; trying from distutils
   FOUND:
     libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
     library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
     language = c
     define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]

lapack_opt_info:
lapack_mkl_info:
mkl_info:
   libraries mkl_ia32,mkl_c_dll,libguide40 not found in 
D:\Programs\Intel\MKL\10.1.0.018\ia32\lib
   NOT AVAILABLE

   NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
   libraries lapack_atlas not found in d:\Docs\ATLAS\build\lib
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
d:\temp\numpy-1.2.1\numpy\distutils\system_info.py:955: UserWarning:
*********************************************************************
     Lapack library (from ATLAS) is probably incomplete:
       size of d:\Docs\ATLAS\build\lib\liblapack.a is 251k (expected >4000k)

     Follow the instructions in the KNOWN PROBLEMS section of the file
     numpy/INSTALL.txt.
*********************************************************************

   warnings.warn(message)
Setting PTATLAS=ATLAS
   FOUND:
     libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
     library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
     language = c

No module named msvccompiler in numpy.distutils; trying from distutils
   FOUND:
     libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
     library_dirs = ['d:\\Docs\\ATLAS\\build\\lib']
     language = c
     define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]

could not resolve pattern in '': '*.txt'
non-existing path in '': 'COMPATIBILITY'
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands 
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands 
--fcompiler options
running build_src
building py_modules sources
building extension "numpy.core.multiarray" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_numpy_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__multiarray_api.h' to sources.
   adding 'build\src.win32-2.5\numpy\core\src' to include_dirs.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\src\\scalartypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\src\\arraytypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__multiarray_api.h']
building extension "numpy.core.umath" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_ufunc_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__ufunc_api.h' to sources.
   adding 'build\src.win32-2.5\numpy\core\src' to include_dirs.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\src\\scalartypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\src\\arraytypes.inc', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__ufunc_api.h']
building extension "numpy.core._sort" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_numpy_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__multiarray_api.h' to sources.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__multiarray_api.h']
building extension "numpy.core.scalarmath" sources
   adding 'build\src.win32-2.5\numpy\core\include/numpy\config.h' to sources.
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\numpyconfig.h' to sources.
executing numpy\core\code_generators\generate_numpy_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__multiarray_api.h' to sources.
executing numpy\core\code_generators\generate_ufunc_api.py
   adding 
'build\src.win32-2.5\numpy\core\include/numpy\__ufunc_api.h' to sources.
numpy.core - nothing done with h_files = 
['build\\src.win32-2.5\\numpy\\core\\include/numpy\\config.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\numpyconfig.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__multiarray_api.h', 
'build\\src.win32-2.5\\numpy\\core\\include/numpy\\__ufunc_api.h']
building extension "numpy.core._dotblas" sources
   adding 'numpy\core\blasdot\_dotblas.c' to sources.
building extension "numpy.lib._compiled_base" sources
building extension "numpy.numarray._capi" sources
building extension "numpy.fft.fftpack_lite" sources
building extension "numpy.linalg.lapack_lite" sources
### Warning:  python_xerbla.c is disabled ###
   adding 'numpy\linalg\lapack_litemodule.c' to sources.
building extension "numpy.random.mtrand" sources
Traceback (most recent call last):
   File "setup.py", line 96, in <module>
     setup_package()
   File "setup.py", line 89, in setup_package
     configuration=configuration )
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\core.py", 
line 184, in setup
   File "d:\programs\python25\lib\distutils\core.py", line 151, in setup
     dist.run_commands()
   File "d:\programs\python25\lib\distutils\dist.py", line 974, in run_commands
     self.run_command(cmd)
   File "d:\programs\python25\lib\distutils\dist.py", line 994, in run_command
     cmd_obj.run()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\install.py", 
line 49, in run
   File "d:\programs\python25\lib\distutils\command\install.py", line 
506, in run
     self.run_command('build')
   File "d:\programs\python25\lib\distutils\cmd.py", line 333, in run_command
     self.distribution.run_command(command)
   File "d:\programs\python25\lib\distutils\dist.py", line 994, in run_command
     cmd_obj.run()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build.py", 
line 37, in run
   File "d:\programs\python25\lib\distutils\command\build.py", line 112, in run
     self.run_command(cmd_name)
   File "d:\programs\python25\lib\distutils\cmd.py", line 333, in run_command
     self.distribution.run_command(command)
   File "d:\programs\python25\lib\distutils\dist.py", line 994, in run_command
     cmd_obj.run()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 130, in run
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 147, in build_sources
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 250, in build_extension_sources
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\build_src.py", 
line 307, in generate_sources
   File "numpy\random\setup.py", line 11, in generate_libraries
     if config_cmd.try_run(tc):
   File "d:\programs\python25\lib\distutils\command\config.py", line 
278, in try_run
     self._check_compiler()
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\command\config.py", 
line 26, in _check_compiler
   File "d:\programs\python25\lib\distutils\command\config.py", line 
107, in _check_compiler
     dry_run=self.dry_run, force=1)
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\ccompiler.py", 
line 366, in new_compiler
   File 
"D:\Programs\Python25\Lib\site-packages\numpy-1.2.1\numpy\distutils\mingw32ccompiler.py", 
line 46, in __init__
   File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 
84, in __init__
     get_versions()
   File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 
424, in get_versions
     ld_version = StrictVersion(result.group(1))
   File "d:\programs\python25\lib\distutils\version.py", line 40, in __init__
     self.parse(vstring)
   File "d:\programs\python25\lib\distutils\version.py", line 107, in parse
     raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '2.18.50.20080625'



From cournape at gmail.com  Sat Jan  3 00:15:53 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 3 Jan 2009 14:15:53 +0900
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103050805.49D18C7C009@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<524D03AE-2D0E-4A9A-8FE7-BBB700BC34D9@mac.com>
	<20090103042733.2F910C7C009@scipy.org>
	<81A59E45-69B7-4351-B44C-95441A5A9F83@mac.com>
	<20090103050805.49D18C7C009@scipy.org>
Message-ID: <5b8d13220901022115p1f07d3desde3ce9a152d1fa06@mail.gmail.com>

On Sat, Jan 3, 2009 at 2:07 PM, Mike Landis <mlandis001 at comcast.net> wrote:
> Does it sort of non-destructively overlay the 2.5 that I'm already running?
>

It only installs numpy into your existing python installation - it
will not overwrite anything else in your python installation (e.g.
everything else should work as before).

You can see numpy as a "plugin" for python, and the binary installer
will just install the numpy "plugin". You will need to have python
installed first.

Note also that if you need more than numpy and scipy, there are also
other options which can make your life easier, like enthought or
pythonxy:

http://www.pythonxy.com/foreword.php
http://www.enthought.com/products/epd.php

In that case, they install many packages related to scientific tasks,
and are self contained. For newcomers on windows, this is the
recommended approach if you don't mind the size.

David


From cournape at gmail.com  Sat Jan  3 00:19:11 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 3 Jan 2009 14:19:11 +0900
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103031048.38829C7C009@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
Message-ID: <5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>

On Sat, Jan 3, 2009 at 12:10 PM, Mike Landis <mlandis001 at comcast.net> wrote:
> Maybe the reason I'm having trouble is that I'm trying to get it
> working on Windows, when almost everyone else is running on Linux?

It is true that most developers use some sort of unix (linux, mac os
X), but we definitely try to make sure it works as well on windows
natively (e.g. without cygwin or any other kind of emulation). But
windows being windows, the only way to make that happen in finite
amount of time is to distribute binaries only (the so called
superpack) - if you are new to python development, you can't expect
being able to build from sources in ten minutes. But you can if you
use the binaries.

David


From cournape at gmail.com  Sat Jan  3 00:24:08 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 3 Jan 2009 14:24:08 +0900
Subject: [Numpy-discussion] installing numpy with Python2.5 already
	present
In-Reply-To: <20090103051153.50F33C7C009@scipy.org>
References: <20090103051153.50F33C7C009@scipy.org>
Message-ID: <5b8d13220901022124p35b94c98u8d5b057877c1193b@mail.gmail.com>

On Sat, Jan 3, 2009 at 2:11 PM, Mike Landis <mlandis001 at comcast.net> wrote:
> This time I included both ATLAS and MKL in the site.cfg file and got
> a little further...  D:\temp\numpy-1.2.1\site.cfg now looks like:
> ...
> ValueError: invalid version number '2.18.50.20080625'

This is a bug in python. Really, you should use the binaries as
recommended previously - it will take one minute, and that's how the
vast majority of windows users install numpy.

David


From mlandis001 at comcast.net  Sat Jan  3 01:07:59 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 01:07:59 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.co
 m>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
Message-ID: <20090103060841.53FA0C8410D@scipy.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090103/b0c1f48c/attachment.html>

From josef.pktd at gmail.com  Sat Jan  3 01:12:30 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 3 Jan 2009 01:12:30 -0500
Subject: [Numpy-discussion] installing numpy with Python2.5 already
	present
In-Reply-To: <5b8d13220901022124p35b94c98u8d5b057877c1193b@mail.gmail.com>
References: <20090103051153.50F33C7C009@scipy.org>
	<5b8d13220901022124p35b94c98u8d5b057877c1193b@mail.gmail.com>
Message-ID: <1cd32cbb0901022212l154cfcecnc6560a78cb2b7702@mail.gmail.com>

On Sat, Jan 3, 2009 at 12:24 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Sat, Jan 3, 2009 at 2:11 PM, Mike Landis <mlandis001 at comcast.net> wrote:
>> This time I included both ATLAS and MKL in the site.cfg file and got
>> a little further...  D:\temp\numpy-1.2.1\site.cfg now looks like:
>> ...
>> ValueError: invalid version number '2.18.50.20080625'
>
> This is a bug in python. Really, you should use the binaries as
> recommended previously - it will take one minute, and that's how the
> vast majority of windows users install numpy.
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

I'm working only on Windows, and I never had any problems with the
numpy installer. And until recently, I never tried to build it myself.
It is easier to install additional packages once numpy, and numpy
distutils are properly set up.

One problem on Windows is to get the path correct, when I'm building
scipy or scikits or any other package with C-extension, I only use
MinGW, and I don't have cygwin installed. On my old computer I had
cygwin, and I read stories that you shouldn't have cygwin on the
Windows path, since some programs got confused with conflicting
Windows and cygwin versions.

Just a guess:
>From your error message, I would think that python distutils finds
your cygwin installation, instead I would hide it by removing it from
the windows path and work only with MingW.


Josef


From josef.pktd at gmail.com  Sat Jan  3 01:30:24 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 3 Jan 2009 01:30:24 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103060841.53FA0C8410D@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
Message-ID: <1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>

On Sat, Jan 3, 2009 at 1:07 AM, Mike Landis <mlandis001 at comcast.net> wrote:
> Cygwin is present, so not just the dumbed down Windows CMD available.
>
> I ran the numpy-1.2.1 superpak.  Verified that it installed (cause you don't
> get near as much output as you do from a shell prompt) by running:
>
>    "
>
> python -c 'import numpy; print numpy.__version__
> '
>
> "
>
> and got the numpy version number back.  Found the site.cfg that the
> installer left in the numpy package directory (it picked up on my ATLAS
> install, but not on MKL) and copied it into d:\temp\scipy-0.7.0b1 and ran
> "python setup.py install" there.  Lots of positive looking output, but it
> ultimately crapped out.  Here's the tail end of that transcript:
>
> copying scipy\weave\__init__.py -> build\lib.win32-2.5\scipy\weave
> running build_clib
> Traceback (most recent call last):
>   File "setup.py", line 92, in <module>
>     setup_package()
>   File "setup.py", line 84, in setup_package
>     configuration=configuration )
>   File "d:\Programs\Python25\lib\site-packages\numpy\distutils\core.py",
> line 184, in setup
>     return old_setup(**new_attr)
>   File "d:\programs\python25\lib\distutils\core.py", line 151, in setup
>     dist.run_commands()
>   File "d:\programs\python25\lib\distutils\dist.py", line 974, in
> run_commands
>     self.run_command(cmd)
>   File "d:\programs\python25\lib\distutils\dist.py", line 994, in
> run_command
>     cmd_obj.run()
>   File
> "d:\Programs\Python25\lib\site-packages\numpy\distutils\command\install.py",
> line 49, in run
>     r = old_install.run(self)
>   File "d:\programs\python25\lib\distutils\command\install.py", line 506, in
> run
>     self.run_command('build')
>   File "d:\programs\python25\lib\distutils\cmd.py", line 333, in run_command
>     self.distribution.run_command(command)
>   File "d:\programs\python25\lib\distutils\dist.py", line 994, in
> run_command
>     cmd_obj.run()
>   File
> "d:\Programs\Python25\lib\site-packages\numpy\distutils\command\build.py",
> line 37, in run
>     old_build.run(self)
>   File "d:\programs\python25\lib\distutils\command\build.py", line 112, in
> run
>     self.run_command(cmd_name)
>   File "d:\programs\python25\lib\distutils\cmd.py", line 333, in run_command
>     self.distribution.run_command(command)
>   File "d:\programs\python25\lib\distutils\dist.py", line 994, in
> run_command
>     cmd_obj.run()
>   File
> "d:\Programs\Python25\lib\site-packages\numpy\distutils\command\build_clib.py",
> line 63, in run
>     force=self.force)
>   File
> "d:\Programs\Python25\lib\site-packages\numpy\distutils\ccompiler.py", line
> 366, in new_compiler
>     compiler = klass(None, dry_run, force)
>   File
> "d:\Programs\Python25\lib\site-packages\numpy\distutils\mingw32ccompiler.py",
> line 46, in __init__
>     verbose,dry_run, force)
>   File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 84, in
> __init__
>     get_versions()
>   File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 424, in
> get_versions
>     ld_version = StrictVersion(result.group(1))
>   File "d:\programs\python25\lib\distutils\version.py", line 40, in __init__
>     self.parse(vstring)
>   File "d:\programs\python25\lib\distutils\version.py", line 107, in parse
>     raise ValueError, "invalid version number '%s'" % vstring
> ValueError: invalid version number '2.18.50.20080625'
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>

If you want to get around this bug, you could correct the version parsing,

change in

File "d:\programs\python25\lib\distutils\cygwinccompiler.py", line 424, in
> get_versions
>     ld_version = StrictVersion(result.group(1))

to

  ld_version = StrictVersion(result.group(1).rsplit('.',1)[0])


see version problem

>>> from distutils.version import StrictVersion
>>> StrictVersion('2.18.50.20080625')
Traceback (most recent call last):
  File "<pyshell#246>", line 1, in <module>
    StrictVersion('2.18.50.20080625')
  File "C:\Programs\Python25\lib\distutils\version.py", line 40, in __init__
    self.parse(vstring)
  File "C:\Programs\Python25\lib\distutils\version.py", line 107, in parse
    raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '2.18.50.20080625'
>>> StrictVersion('2.18.50.20080625'.rsplit('.',1)[0])
StrictVersion ('2.18.50')



But, see my other message, try without cygwin. I usually don't have
any problems building scipy with mingw.

Josef


From david at ar.media.kyoto-u.ac.jp  Sat Jan  3 01:56:27 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 03 Jan 2009 15:56:27 +0900
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103060841.53FA0C8410D@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>	<20090103031048.38829C7C009@scipy.org>	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
Message-ID: <495F0C1B.5020407@ar.media.kyoto-u.ac.jp>

Mike Landis wrote:
> Cygwin is present, so not just the dumbed down Windows CMD available.

You should not use cygwin: if you use cygwin, it will build numpy
against the cygwin python, or worse, will be very confused, because you
will mix cygwin compilers and mingw compilers.  Unless you want to build
numpy for cygwin python, you should not try anything from cygwin. Not
that it is impossible, but there are many warts to avoid, and it does
not worth the pain unless you are a numpy developer.

>
> I ran the numpy-1.2.1 superpak.  Verified that it installed (cause you
> don't get near as much output as you do from a shell prompt) by running:
>
>    "
> python -c 'import numpy; print numpy.__version__
> '
> "

See, it is easier  :)

>
> and got the numpy version number back.  Found the site.cfg that the
> installer left in the numpy package directory (it picked up on my
> ATLAS install, but not on MKL) and copied it into
> d:\temp\scipy-0.7.0b1 and ran "python setup.py install" there.

This will not work: you need blas/lapack for scipy - the site.cfg in
installed numpy refers to a blas/lapack which was used when I built the
binary installer - and it not installed (blas/lapack is statically
linked, as dynamically linked libraries is too difficult on windows).
Please install from the scipy binary installer instead:

http://sourceforge.net/project/showfiles.php?group_id=27747

Trust me, it will take you less time.

> Lots of positive looking output, but it ultimately crapped out. 
> Here's the tail end of that transcript:

Same python bug as before: the problem is that for some reason, python
checks the version of your toolchain (here binutils), and refuses to
build when the version is not the one expected. The easiest fix is to
follow Joseph suggestion. But again, you need a blas/lapack first.

David


From mlandis001 at comcast.net  Sat Jan  3 10:13:55 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 10:13:55 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.co
 m>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
Message-ID: <20090103151440.EBE93C8410D@scipy.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090103/6d5897de/attachment.html>

From josef.pktd at gmail.com  Sat Jan  3 10:50:42 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 3 Jan 2009 10:50:42 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103151440.EBE93C8410D@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
Message-ID: <1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>

On Sat, Jan 3, 2009 at 10:13 AM, Mike Landis <mlandis001 at comcast.net> wrote:
> I do not have cygwin in my windows path, so I guess that's already hidden.
>
> I patched  d:\programs\python25\lib\distutils\cygwinccompiler.py, line 424
> to read
>
> ld_version = StrictVersion(result.group(1).rsplit('.',1)[0])
>
> but I still got crash and a traceback.
>
> David Cournapeau suggested using the scipy superpack, so i tried
> scipy-0.7.0b1-win32-superpack-python2.5.exe.  If there were errors I
> wouldn't know about them, but running
>
> 'python -c 'import scipy; print scipy.__version__ '
>
> produces a version number (0.7.0.dev5180 ... not exactly the 0.7.0b1 you'd
> expect, but not a stack trace either).
>
> Now all I have to do is find some test cases so I can verify a little deeper
> than the version number.
>

If you have nose installed, you can run the scipy test suite with

import scipy
scipy.test()

Josef


From pgmdevlist at gmail.com  Sat Jan  3 15:26:14 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sat, 3 Jan 2009 15:26:14 -0500
Subject: [Numpy-discussion] genloadtxt : ready for inclusion
Message-ID: <2FA139CE-ACC2-4632-82EB-90627196AFA6@gmail.com>

All,
You'll probably remember that last December, I started rewriting  
np.loadtxt and ame up with a series of functions that support missing  
data. I tried to copy/paste the code in numpy.lib.io.py but ran into  
dependency problems and left it at that. I think that part of the  
reason is that the code relies on numpy.ma which can't be loaded when  
numpy.lib gets loaded.

As I needed a way to grant access to the code to anybody, I created a  
small project on launchpad: you can access it to:

https://code.launchpad.net/~pierregm/numpy/numpy_addons

The loadtxt reimplementation functions can be found in the  
numpy.io.fromascii module, their unittest in the corresponding test  
directory. In addition, you'll find several other functions and their  
unittest to manipulate arrays w/ flexible data-type. They are  
basically rewritten version of some functions in matplotlib.mlab.

Would anybody be willing to try inserting the new functions in numpy ?  
I was hoping the genfromtxt and consorts would make it to numpy 1.3.x  
(I'd need the code for the scikits.timeseries package).

As usual, I'd need all the feedback you can share.

Thanks a lot in advance.
P.


From mlandis001 at comcast.net  Sat Jan  3 17:27:30 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 17:27:30 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.co
 m>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
Message-ID: <20090103222816.5B76EC7C026@scipy.org>

Thanks for the suggestion Joseph - the scipy test suite runs, but it 
produces lots of errors.

Some deprecation warnings in numpy\lib\utils.py (line 110) and 
scipy\linalg\decomp.py (line 1173)

Than it complains about a '_bad_path_' (doesn't exist or not 
writable).  Couldn't remove 
<user>\appdata]local\temp\tmpc192_pcat_test (directory not empty)

Then the crap hits the fan...  the following crashes are all due to: 
test_polyint.CheckKrogh
test_derivative crashes on line 38 of scipy\interpolate\tests\test_polyint.py
test_derivatives crashes on line 26
test_empty crashes in the same file on line 73
test_hermite crashes in the same file on line 57
test_high_derivative crashes in the same file on line 44
test_lagrange crashes in the same file on line 19
test_low_derivatives crashes in the same file line 32
test_scalar on line 22
test_shapes_1d_vectorvalue on line 95
test_shapes_scalarvalue on line 76
test_shapes_scalarvalue_derivative on line 82
test_shapes_vectorvalue on line 89
test_shapes_vectorvalue_derivative on line 101
test_vector on line 63
test_wrapper on line 108

The following crashes are due to test_polyint.CheckPiecewise
test_construction on line 186
test_derivative on line 193
test_derivatives on line 196
test_incremental on line 217
test_scalar on line 189
test_shapes_scalarvalue on line 221
test_shapes_scalarvalue_derivative on line 227
test_shapes_vectorvalue on line 235
test_shapes_vectorvalue_1d on line 242
test_shapes_vectorvalue_derivative on line 248
test_vector on line 205
test_wrapper on line 255

The following crashes are due to test_polyint.CheckTaylor
test_exponential on line 116

Failure: AttributeError ('module' object has no attribute 'byteordercodes')
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name numpyio)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name fblas)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name flapack)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name fblas)  AGAIN
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name flapack)  AGAIN
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: NameError (name 'pilutil is not defined)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name cobyla)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name nonlin)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name zeros)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: NameError (name 'pilutil is not defined)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: ImportError (cannot import name linsolve)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: AttributeError ('module' object has no attribute '_cephes')
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: NameError (name 'pilutil is not defined)
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

Failure: AttributeError ('module' object has no attribute 'convolve')
nose-0.10.30py2.5.egg\nose\loader.py on line 364 in loadTestsFromName

That was with nose-0.10.3-py2.5.  I upgrades to the latest 
nose-0.10.4-py2.5, but it still produced an armload of (what look 
like the same) errors.

Does this symptomology point to anything (configuration error, 
package out of date, ???)

At 10:50 AM 1/3/2009, you wrote:
>On Sat, Jan 3, 2009 at 10:13 AM, Mike Landis <mlandis001 at comcast.net> wrote:
> > I do not have cygwin in my windows path, so I guess that's already hidden.
> >
> > I patched  d:\programs\python25\lib\distutils\cygwinccompiler.py, line 424
> > to read
> >
> > ld_version = StrictVersion(result.group(1).rsplit('.',1)[0])
> >
> > but I still got crash and a traceback.
> >
> > David Cournapeau suggested using the scipy superpack, so i tried
> > scipy-0.7.0b1-win32-superpack-python2.5.exe.  If there were errors I
> > wouldn't know about them, but running
> >
> > 'python -c 'import scipy; print scipy.__version__ '
> >
> > produces a version number (0.7.0.dev5180 ... not exactly the 0.7.0b1 you'd
> > expect, but not a stack trace either).
> >
> > Now all I have to do is find some test cases so I can verify a 
> little deeper
> > than the version number.
> >
>
>If you have nose installed, you can run the scipy test suite with
>
>import scipy
>scipy.test()
>
>Josef
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at scipy.org
>http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>No virus found in this incoming message.
>Checked by AVG - http://www.avg.com
>Version: 8.0.176 / Virus Database: 270.10.2/1872 - Release Date: 
>1/2/2009 1:10 PM



From mlandis001 at comcast.net  Sat Jan  3 21:56:39 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 21:56:39 -0500
Subject: [Numpy-discussion] debugging scipy install
Message-ID: <20090104025726.093E0C7C026@scipy.org>

running "python -c 'import scipy; scipy.test()' "
finding one of the earliest errors referencing a bad_path, I found 
scipy's INSTALL.txt, paragraph 5, under the TROUBLESHOOTING says to 
"cd scipy/Lib/linalg" so you can run:
'python setup_atlas_version.py build_ext --inplace --force'

[ setup_atlas_version.py is actually in scipy/linalg (at least on Windows) ]

...  but that shell command bombs out with the following Traceback
   File "setup_atlas_version.py", line 7 in <module>
     from numpy.distutils.misc_util_import get_path, default_config_dict
ImportError: cannot import name get_path

Seems like getting the path could be source of a '_bad_path_' problem.

Does anyone know where get_path is defined, and if so, where the file 
is downloadable from?

After taking the superpack install routes with both numpy and scipy, 
shouldn't these issues have been taken care of already?



From josef.pktd at gmail.com  Sat Jan  3 22:40:28 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 3 Jan 2009 22:40:28 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090103222816.5B76EC7C026@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
Message-ID: <1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>

Make sure that when you import scipy that you get the correct version.

>>> import scipy
>>> scipy.__file__
'C:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'

>From your error messages, I would think python is loading the source
distribution and not the compiled and installed version. It would be
helpful to see your actual error messages from nose, with copy and
paste, at least the first few and last parts of the nose tests. Your
summary error message is not very helpful because it doesn't show your
actual error path and trace backs.

When I installed the 0.7.0 b1 superpack on WindowsXP, it worked out of
the box. The only thing to do, before installing a new version of
numpy or scipy, is to uninstall or delete any old version in
site-packages, since the directory names of scipy and numpy do not
include version numbers. Installing on top of an old version, can
leave some old files around which sometimes cause errors.


Josef


From mlandis001 at comcast.net  Sat Jan  3 23:20:40 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 23:20:40 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.co
 m>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
Message-ID: <20090104042129.8D284C7C026@scipy.org>

Josef (sorry about spelling your name wrong in a previous post),

Thanks for the continued suggestions.  I deleted the site-packages: 
numpy and scipy, and reinstalled each using the current release 
superpacks (numpy first, then scipy).

then I ran:

python -c 'import numpy; numpy.test()'

and got:

Running unit tests for numpy
NumPy version 1.2.1
NumPy is installed in d::\Programs\Python25\lib\site-packages\numpy
Python version 2.5.2 (r252:60911, Mar 27 2008, 17:57:18) IMSC v.1310 
32 bit (ntel)]
nose version 0.10.4
Traceback (most recent call last):
   File "(string)", line 1, in <module>
   File 
"d::\Programs\Python25\lib\site-packages\numpy\testing\nosetester.py", 
line 264, in test
     import doctest
   File"d:\Programs\Python25\lib\doctest.p", line 99, in <module>
     import unittest, difflib, pd, tempfile
   File "d:\programs\python25\lib\tempfile.py", line 33, in <module>
     from random import Random as __Random
ImportError: cannot import name Random

scipy.test() bombs out with the exact same Traceback, except that it 
mentions the scipy version (0,7.0.dev5180) and scipy install 
directory just before it echoes the line with the python version number.

So, it's not getting as far as it was with the source mixed in.

At 10:40 PM 1/3/2009, you wrote:
>Make sure that when you import scipy that you get the correct version.
>
> >>> import scipy
> >>> scipy.__file__
>'C:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'
>
> >From your error messages, I would think python is loading the source
>distribution and not the compiled and installed version. It would be
>helpful to see your actual error messages from nose, with copy and
>paste, at least the first few and last parts of the nose tests. Your
>summary error message is not very helpful because it doesn't show your
>actual error path and trace backs.
>
>When I installed the 0.7.0 b1 superpack on WindowsXP, it worked out of
>the box. The only thing to do, before installing a new version of
>numpy or scipy, is to uninstall or delete any old version in
>site-packages, since the directory names of scipy and numpy do not
>include version numbers. Installing on top of an old version, can
>leave some old files around which sometimes cause errors.
>
>
>Josef
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at scipy.org
>http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Sat Jan  3 23:29:26 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 3 Jan 2009 22:29:26 -0600
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090104042129.8D284C7C026@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
	<20090104042129.8D284C7C026@scipy.org>
Message-ID: <3d375d730901032029y7a2b2820m908ce2748384663e@mail.gmail.com>

On Sat, Jan 3, 2009 at 22:20, Mike Landis <mlandis001 at comcast.net> wrote:
> Josef (sorry about spelling your name wrong in a previous post),
>
> Thanks for the continued suggestions.  I deleted the site-packages:
> numpy and scipy, and reinstalled each using the current release
> superpacks (numpy first, then scipy).
>
> then I ran:
>
> python -c 'import numpy; numpy.test()'
>
> and got:
>
> Running unit tests for numpy
> NumPy version 1.2.1
> NumPy is installed in d::\Programs\Python25\lib\site-packages\numpy
> Python version 2.5.2 (r252:60911, Mar 27 2008, 17:57:18) IMSC v.1310
> 32 bit (ntel)]
> nose version 0.10.4
> Traceback (most recent call last):
>   File "(string)", line 1, in <module>
>   File
> "d::\Programs\Python25\lib\site-packages\numpy\testing\nosetester.py",
> line 264, in test
>     import doctest
>   File"d:\Programs\Python25\lib\doctest.p", line 99, in <module>
>     import unittest, difflib, pd, tempfile
>   File "d:\programs\python25\lib\tempfile.py", line 33, in <module>
>     from random import Random as __Random
> ImportError: cannot import name Random

What directory are you in? I'm guessing that you are in a numpy/
directory either in the source tree or under site-packages/. Change
out of that directory. Python looks for modules in the current
directory before the standard locations, so it's picking up the
numpy.random subpackage instead of the standard library module random.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From mlandis001 at comcast.net  Sat Jan  3 23:38:17 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sat, 03 Jan 2009 23:38:17 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.co
 m>
References: <20090102183005.27311C7C019@scipy.org>
	<e76aa17f0901021045x2171441i50584df8c6a14607@mail.gmail.com>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
Message-ID: <20090104043904.7CEFCC7C029@scipy.org>

Maybe the following will also be useful...  Recall that I completely 
deleted numpy and scipy and reinstalled each from their respective 
superpacks, then ran:

 >>> import numpy; numpy.__file__
'D:\\Programs\\Python25\\lib\\site-packages\\numpy\\__init__.pyc'

 >>> import scipy; scipy.__file__
'D:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'

$ cd site-packages\numpy; python setup.py config
This is the wrong setup.py file to run

$ cd site-packages\scipy; python setup.py config
non-existing path in 'cluster': 'src\\vq_module.c'
non-existing path in 'cluster': 'src\\vq.c'
non-existing path in 'cluster': 'src\\hierarchy_wrap.c'
non-existing path in 'cluster': 'src\\hierarchy.c'
Appending scipy.cluster configuration to scipy
Ignoring attempt to set 'name' (from 'scipy' to 'scipy.cluster')
Warning: No configuration returned, assuming unavailable.
Appending scipy.constants configuration to scipy
Ignoring attempt to set 'name' (from 'scipy' to 'scipy.constants')
could not resolve pattern in 'fftpack': 'src/dfftpack\\*.f'
non-existing path in 'fftpack': 'fftpack.pyf'
non-existing path in 'fftpack': 'src/zfft.c'
non-existing path in 'fftpack': 'src/drfft.c'
non-existing path in 'fftpack': 'src/zrfft.c'
non-existing path in 'fftpack': 'src/zfftnd.c'
non-existing path in 'fftpack': 'src/zfft_fftpack.c'
non-existing path in 'fftpack': 'src/drfft_fftpack.c'
non-existing path in 'fftpack': 'src/zfftnd_fftpack.c'
non-existing path in 'fftpack': 'convolve.pyf'
non-existing path in 'fftpack': 'src/convolve.c'
Appending scipy.fftpack configuration to scipy
Ignoring attempt to set 'name' (from 'scipy' to 'scipy.fftpack')
d:\Programs\Python25\lib\site-packages\numpy\distutils\system_info.py:1340: 
UserWarning:
     Atlas (http://math-atlas.sourceforge.net/) libraries not found.
     Directories to search for the libraries can be specified in the
     numpy/distutils/site.cfg file (section [atlas]) or by setting
     the ATLAS environment variable.
   warnings.warn(AtlasNotFoundError.__doc__)
d:\Programs\Python25\lib\site-packages\numpy\distutils\system_info.py:1349: 
UserWarning:
     Blas (http://www.netlib.org/blas/) libraries not found.
     Directories to search for the libraries can be specified in the
     numpy/distutils/site.cfg file (section [blas]) or by setting
     the BLAS environment variable.
   warnings.warn(BlasNotFoundError.__doc__)
d:\Programs\Python25\lib\site-packages\numpy\distutils\system_info.py:1352: 
UserWarning:
     Blas (http://www.netlib.org/blas/) sources not found.
     Directories to search for the sources can be specified in the
     numpy/distutils/site.cfg file (section [blas_src]) or by setting
     the BLAS_SRC environment variable.
   warnings.warn(BlasSrcNotFoundError.__doc__)
Traceback (most recent call last):
   File "setup.py", line 32, in <module>
     setup(**configuration(top_path='').todict())
   File "setup.py", line 8, in configuration
     config.add_subpackage('integrate')
   File 
"d:\Programs\Python25\lib\site-packages\numpy\distutils\misc_util.py", 
line 851, in add_subpackage
     caller_level = 2)
   File 
"d:\Programs\Python25\lib\site-packages\numpy\distutils\misc_util.py", 
line 834, in get_subpackage
     caller_level = caller_level + 1)
   File 
"d:\Programs\Python25\lib\site-packages\numpy\distutils\misc_util.py", 
line 781, in _get_configuration_from_setup_py
     config = setup_module.configuration(*args)
   File 
"D:\Programs\Python25\Lib\site-packages\scipy\integrate\setup.py", 
line 10, in configuration
     blas_opt = get_info('blas_opt',notfound_action=2)
   File 
"d:\Programs\Python25\lib\site-packages\numpy\distutils\system_info.py", 
line 267, in get_info
     return cl().get_info(notfound_action)
   File 
"d:\Programs\Python25\lib\site-packages\numpy\distutils\system_info.py", 
line 416, in get_info
     raise self.notfounderror,self.notfounderror.__doc__
numpy.distutils.system_info.BlasNotFoundError:
     Blas (http://www.netlib.org/blas/) libraries not found.
     Directories to search for the libraries can be specified in the
     numpy/distutils/site.cfg file (section [blas]) or by setting
     the BLAS environment variable.

At 10:40 PM 1/3/2009, you wrote:
>Make sure that when you import scipy that you get the correct version.
>
> >>> import scipy
> >>> scipy.__file__
>'C:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'
>
> >From your error messages, I would think python is loading the source
>distribution and not the compiled and installed version. It would be
>helpful to see your actual error messages from nose, with copy and
>paste, at least the first few and last parts of the nose tests. Your
>summary error message is not very helpful because it doesn't show your
>actual error path and trace backs.
>
>When I installed the 0.7.0 b1 superpack on WindowsXP, it worked out of
>the box. The only thing to do, before installing a new version of
>numpy or scipy, is to uninstall or delete any old version in
>site-packages, since the directory names of scipy and numpy do not
>include version numbers. Installing on top of an old version, can
>leave some old files around which sometimes cause errors.
>
>
>Josef
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at scipy.org
>http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Sat Jan  3 23:42:52 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 3 Jan 2009 22:42:52 -0600
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090104043904.7CEFCC7C029@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
	<20090104043904.7CEFCC7C029@scipy.org>
Message-ID: <3d375d730901032042t4b123fcbu221f254c8798ad54@mail.gmail.com>

On Sat, Jan 3, 2009 at 22:38, Mike Landis <mlandis001 at comcast.net> wrote:
> Maybe the following will also be useful...  Recall that I completely
> deleted numpy and scipy and reinstalled each from their respective
> superpacks, then ran:
>
>  >>> import numpy; numpy.__file__
> 'D:\\Programs\\Python25\\lib\\site-packages\\numpy\\__init__.pyc'
>
>  >>> import scipy; scipy.__file__
> 'D:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'
>
> $ cd site-packages\numpy; python setup.py config
> This is the wrong setup.py file to run

That message is correct. Don't do that. You don't run the setup.py
scripts on installed binaries.

> $ cd site-packages\scipy; python setup.py config

Don't do that either.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Sat Jan  3 23:52:46 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 3 Jan 2009 23:52:46 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <3d375d730901032042t4b123fcbu221f254c8798ad54@mail.gmail.com>
References: <20090102183005.27311C7C019@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
	<20090104043904.7CEFCC7C029@scipy.org>
	<3d375d730901032042t4b123fcbu221f254c8798ad54@mail.gmail.com>
Message-ID: <1cd32cbb0901032052v73067cd9la64dcea8f53ed562@mail.gmail.com>

On Sat, Jan 3, 2009 at 11:42 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sat, Jan 3, 2009 at 22:38, Mike Landis <mlandis001 at comcast.net> wrote:
>> Maybe the following will also be useful...  Recall that I completely
>> deleted numpy and scipy and reinstalled each from their respective
>> superpacks, then ran:
>>
>>  >>> import numpy; numpy.__file__
>> 'D:\\Programs\\Python25\\lib\\site-packages\\numpy\\__init__.pyc'
>>
>>  >>> import scipy; scipy.__file__
>> 'D:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'
>>
>> $ cd site-packages\numpy; python setup.py config
>> This is the wrong setup.py file to run
>
> That message is correct. Don't do that. You don't run the setup.py
> scripts on installed binaries.
>
>> $ cd site-packages\scipy; python setup.py config
>
> Don't do that either.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

Mike,
don't go in site-packages at all, except for browsing and looking at
the source for information.

basic steps:

run installer
go to a working directory, not under the python directory in programs,
(and without any scipy, numpy source in it)
then start python or idle there
and then import numpy and scipy and test, that's it.

Unless there is a reason for you to change the numpy, scipy source
there is no reason for you to touch any of the config, compile build
steps. They are for later, when you need additional packages that
don't have an installer.

To get up and running, I recommend just to follow the basic steps for a user.

Josef


From cournape at gmail.com  Sat Jan  3 23:55:47 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 4 Jan 2009 13:55:47 +0900
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090104043904.7CEFCC7C029@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
	<20090104043904.7CEFCC7C029@scipy.org>
Message-ID: <5b8d13220901032055tfe9839w3386d79659ffc852@mail.gmail.com>

On Sun, Jan 4, 2009 at 1:38 PM, Mike Landis <mlandis001 at comcast.net> wrote:
> Maybe the following will also be useful...  Recall that I completely
> deleted numpy and scipy and reinstalled each from their respective
> superpacks, then ran:
>
>  >>> import numpy; numpy.__file__
> 'D:\\Programs\\Python25\\lib\\site-packages\\numpy\\__init__.pyc'
>
>  >>> import scipy; scipy.__file__
> 'D:\\Programs\\Python25\\lib\\site-packages\\scipy\\__init__.pyc'
>

At this stage, you're *done*. Everything is installed, you don't have
to do anything anymore. I am sorry if the following is obvious, but
that's the only explanation I can make: there are two ways to install
open source softwares - from sources, or from binary installers. By
using the superpack, you are using the later - using setup.py implied
the former. So what you end up doing is to try building the software
from the binary - which does not make sense.

It may not look like it at this point, but you are making things much
more complicate than they really are :) After the super pack
executions, you have run the installers successfully, so everything is
installed, without any further step to follow.

cheers,

David


From mlandis001 at comcast.net  Sun Jan  4 00:15:18 2009
From: mlandis001 at comcast.net (Mike Landis)
Date: Sun, 04 Jan 2009 00:15:18 -0500
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <3d375d730901032029y7a2b2820m908ce2748384663e@mail.gmail.co
 m>
References: <20090102183005.27311C7C019@scipy.org>
	<20090103031048.38829C7C009@scipy.org>
	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>
	<20090103060841.53FA0C8410D@scipy.org>
	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>
	<20090103151440.EBE93C8410D@scipy.org>
	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>
	<20090103222816.5B76EC7C026@scipy.org>
	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>
	<20090104042129.8D284C7C026@scipy.org>
	<3d375d730901032029y7a2b2820m908ce2748384663e@mail.gmail.com>
Message-ID: <20090104051606.758A9C84112@scipy.org>

I cd'd out of numpy and site-packages and re-ran the package 
tests.  both numpy.test() and scipy.test() ran without serious 
errors.  Some DeprecationWarnings and integrals that are probably 
divergent or slowly convergent...

It's looking much more promising.  Two gotchas on top of each other - 
not deleting the previously existing source, and running scripts from 
within the site-packages directory.

Whew!  numpy/scipy was looking scarily unstable until discovering my 
install and pilot errors.

Thanks for all of the help ... 



From david at ar.media.kyoto-u.ac.jp  Sun Jan  4 02:15:45 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 04 Jan 2009 16:15:45 +0900
Subject: [Numpy-discussion] building numpy/scipy
In-Reply-To: <20090104051606.758A9C84112@scipy.org>
References: <20090102183005.27311C7C019@scipy.org>	<20090103031048.38829C7C009@scipy.org>	<5b8d13220901022119v3ef22a3bw3f8aee29caee600b@mail.gmail.com>	<20090103060841.53FA0C8410D@scipy.org>	<1cd32cbb0901022230x2c56adc7tf6a3de4bd478881e@mail.gmail.com>	<20090103151440.EBE93C8410D@scipy.org>	<1cd32cbb0901030750s6745d840n64cd7c916f5a77d4@mail.gmail.com>	<20090103222816.5B76EC7C026@scipy.org>	<1cd32cbb0901031940x565d925eo1e4037ebfe7caa5b@mail.gmail.com>	<20090104042129.8D284C7C026@scipy.org>	<3d375d730901032029y7a2b2820m908ce2748384663e@mail.gmail.com>
	<20090104051606.758A9C84112@scipy.org>
Message-ID: <49606221.8080101@ar.media.kyoto-u.ac.jp>

Mike Landis wrote:
> I cd'd out of numpy and site-packages and re-ran the package 
> tests.  both numpy.test() and scipy.test() ran without serious 
> errors.  Some DeprecationWarnings and integrals that are probably 
> divergent or slowly convergent...
>
> It's looking much more promising.  Two gotchas on top of each other - 
> not deleting the previously existing source, and running scripts from 
> within the site-packages directory.
>
> Whew!  numpy/scipy was looking scarily unstable until discovering my 
> install and pilot errors.
>   

Glad it is working for you. We are sorry about the installation issues:
we know things could be better, and we hope to improve things on that
front - they have improved quite a bit already.

cheers,

David


From pgmdevlist at gmail.com  Sun Jan  4 16:44:33 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 4 Jan 2009 16:44:33 -0500
Subject: [Numpy-discussion] unique1d and asarray
Message-ID: <FAB3FF41-98C5-4471-86B0-B876A7672168@gmail.com>

All,
Currently, np.unique1d uses np.asarray to ensure the input is an  
array. The problem is that np.asarray transforms a MaskedArray into a  
regular ndarray, the missing information is lost and the result is not  
correct.
If we used np.asanyarray instead, subclasses are recognized properly,  
the mask is recognized by argsort and the result correct.
Is there a reason why we use np.asarray instead of np.asanyarray ?
Thanks a lot in advance,
P.


From robert.kern at gmail.com  Sun Jan  4 16:47:37 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 4 Jan 2009 15:47:37 -0600
Subject: [Numpy-discussion] unique1d and asarray
In-Reply-To: <FAB3FF41-98C5-4471-86B0-B876A7672168@gmail.com>
References: <FAB3FF41-98C5-4471-86B0-B876A7672168@gmail.com>
Message-ID: <3d375d730901041347j72f4dba2p36dd3a03ccade2ae@mail.gmail.com>

On Sun, Jan 4, 2009 at 15:44, Pierre GM <pgmdevlist at gmail.com> wrote:
> All,
> Currently, np.unique1d uses np.asarray to ensure the input is an
> array. The problem is that np.asarray transforms a MaskedArray into a
> regular ndarray, the missing information is lost and the result is not
> correct.
> If we used np.asanyarray instead, subclasses are recognized properly,
> the mask is recognized by argsort and the result correct.
> Is there a reason why we use np.asarray instead of np.asanyarray ?

Probably not.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Sun Jan  4 16:51:48 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 4 Jan 2009 16:51:48 -0500
Subject: [Numpy-discussion] unique1d and asarray
In-Reply-To: <3d375d730901041347j72f4dba2p36dd3a03ccade2ae@mail.gmail.com>
References: <FAB3FF41-98C5-4471-86B0-B876A7672168@gmail.com>
	<3d375d730901041347j72f4dba2p36dd3a03ccade2ae@mail.gmail.com>
Message-ID: <2983337C-89AE-4FE6-8DAB-A68DA04F0380@gmail.com>


On Jan 4, 2009, at 4:47 PM, Robert Kern wrote:

> On Sun, Jan 4, 2009 at 15:44, Pierre GM <pgmdevlist at gmail.com> wrote:
>>
>> If we used np.asanyarray instead, subclasses are recognized properly,
>> the mask is recognized by argsort and the result correct.
>> Is there a reason why we use np.asarray instead of np.asanyarray ?
>
> Probably not.

So there wouldn't be any objections to make the switch ? We can wait a  
couple of days if anybody has a pb with that...


From ghzhao6917 at yahoo.com.cn  Sun Jan  4 20:08:21 2009
From: ghzhao6917 at yahoo.com.cn (gh zhao)
Date: Mon, 5 Jan 2009 09:08:21 +0800 (CST)
Subject: [Numpy-discussion] cobyla, f2py
Message-ID: <552189.73120.qm@web15605.mail.cnb.yahoo.com>

?just?try ?f2py according to http://www.scipy.org/F2PY_Windows.
?The following ?woks in dos? cmd window.
E:\Miser3p2F\opcntrl\python E:\python25\scripts\f2py.py -c --fcompiler=gnu95 --compiler=mingw32 -lmsvcr71 -m cobyla cobyla.pyf cobyla2.f trstlp.f 

But I directly run run_compile() of? f2py2e with 
??? sys.argv=sys.argv+['-c','--fcompiler=gnu95','--compiler=mingw32', '-lmsvcr71','-m', 'cobyla','E:\\Miser3p2F\opcntrl\\cobyla.pyf','E:\\Miser3p2F\opcntrl\\cobyla2.f','E:\\Miser3p2F\opcntrl\\trstlp.f'] 
, it does not work. 
I have tried several simple examples, they all work.
I do not know why.
?
ghzhao from Curtin


      ___________________________________________________________ 
  ????????????????? 
http://card.mail.cn.yahoo.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090105/474be074/attachment.html>

From zhangyunfeng at gmail.com  Mon Jan  5 01:55:46 2009
From: zhangyunfeng at gmail.com (Yunfeng Zhang)
Date: Sun, 4 Jan 2009 22:55:46 -0800 (PST)
Subject: [Numpy-discussion] Invitation to connect on LinkedIn
Message-ID: <149989302.5199552.1231138546697.JavaMail.app@ech3-cdn06.prod>

LinkedIn
------------



   
Discussion,

I'd like to add you to my professional network on LinkedIn.

- Yunfeng

Learn more:
https://www.linkedin.com/e/isd/443568042/FmoYJeHZ/

------------------------------------------

What is LinkedIn and why should you join?
http://learn.linkedin.com/what-is-linkedin/


 
------
(c) 2008, LinkedIn Corporation

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090104/2219dc39/attachment.html>

From nadavh at visionsense.com  Mon Jan  5 02:27:10 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Mon, 5 Jan 2009 09:27:10 +0200
Subject: [Numpy-discussion] Zoom fft code
Message-ID: <710F2847B0018641891D9A216027636029C39B@ex3.envision.co.il>

 I am looking for a zoom fft code. I found an old code by Paule Kinzle (a matlab code with a translation to numarray), but its 2D extension (czt1.py) looks buggy.

  Nadav.




From stefan at sun.ac.za  Mon Jan  5 03:25:57 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 5 Jan 2009 10:25:57 +0200
Subject: [Numpy-discussion] Zoom fft code
In-Reply-To: <710F2847B0018641891D9A216027636029C39B@ex3.envision.co.il>
References: <710F2847B0018641891D9A216027636029C39B@ex3.envision.co.il>
Message-ID: <9457e7c80901050025j520ce5dds26d914d63e332e76@mail.gmail.com>

Hi Nadav

I recall that you posted an implementation yourself a while ago!

http://www.mail-archive.com/numpy-discussion at scipy.org/msg01812.html

Regards
St?fan

2009/1/5 Nadav Horesh <nadavh at visionsense.com>:
>
>  I am looking for a zoom fft code. I found an old code by Paule Kinzle (a matlab code with a translation to numarray), but its 2D extension (czt1.py) looks buggy.
>
>  Nadav.


From ezindy at gmail.com  Mon Jan  5 04:42:31 2009
From: ezindy at gmail.com (Egor Zindy)
Date: Mon, 5 Jan 2009 18:42:31 +0900
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
Message-ID: <c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>

Hello Rich,

sorry it took so long to answer back, holidays and all :-)

That's exactly the kind of SWIG / numpy.i problems I've been working on over
the past few months: How to generate an array you don't know the size of
a-priori, and then handle the memory deallocation seamlessly. In your case,
you know that the output array will be half the size of the input array, but
this falls under the more general case of "not knowing the output size
a-priori".

Have a look at the files attached. I've rewritten your function header as:
void sms_spectrumMag( int sizeInMag, float *pInRect, int *sizeOutMag, float
**pOutMag);

Easy to see what the input and output arrays are now. Then my numpy.i
handles the memory deallocation of the **pOutMag array.

I've actually moved my numpy.i explanations to the scipy/numpy cookbook last
week :-)
http://www.scipy.org/Cookbook/SWIG_Memory_Deallocation

Hope it all makes sense. If you have any questions, don't hesitate!

>python test_dftmagnitude.py
[1, 1, 2, 2]
[ 1.41421354  2.82842708]
[1, 1, 2, 2, 3, 3, 4, 4]
[ 1.41421354  2.82842708  4.2426405   5.65685415]
[1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
[ 1.41421354  2.82842708  4.2426405   5.65685415  7.07106781]

Regards,
Egor

On Wed, Dec 24, 2008 at 1:52 AM, Rich E <reakinator at gmail.com> wrote:

> Hi list,
>
> My question has to do with the Numpy/SWIG typemapping system.
>
> I recently got the typemaps in numpy.i to work on most of my C
> functions that are wrapped using SWIG, if they have arguments of the
> form (int sizeArray, float *pArray).
>
> Now I am trying to figure out how to wrap function that aren't of the
> form, such as the following function:
>
> /*! \brief compute magnitude spectrum of a DFT
>  *
>  * \param sizeMag              size of output Magnitude (half of input real
> FFT)
>  * \param pFReal               pointer to input FFT real array
> (real/imag floats)
>  * \param pFMAg        pointer to float array of magnitude spectrum
>  */
> void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag)
> {
>       int i, it2;
>       float fReal, fImag;
>
>       for (i=0; i<sizeMag; i++)
>       {
>               it2 = i << 1;
>               fReal = pInRect[it2];
>               fImag = pInRect[it2+1];
>               pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag);
>       }
> }
>
> There are two arrays, one is half the size of the other.  But, SWIG
> doesn't know this, according to the type map it will think *pInRect is
> of size sizeMag and will not know anything about *pOutMag.
>
> Ideally in python, I would like to call the function as
> sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of
> nArray2, and nArray2 is of size sizeMag.
>
> I think in order to do this (although if someone has a better
> suggestion, I am open to it), I will have to modify the typemap in
> order to tell SWIG how to call the C function properly.  I do not want
> to have to edit the wrapped C file every time it is regenerated from
> the interface file.
>
>
> Here is a start I made with the existing typemap code in numpy.i (not
> working):
>
> /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>  */
> %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY,
>          fragment="NumPy_Macros")
>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
> {
>  $1 = is_array($input) && PyArray_EquivTypenums(array_type($input),
>                                                DATA_TYPECODE);
> }
> %typemap(in,
>        fragment="NumPy_Fragments")
>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>  (PyArrayObject* array=NULL, int i=0)
> {
>  array = obj_to_array_no_conversion($input, DATA_TYPECODE);
>  if (!array || !require_dimensions(array,1) || !require_contiguous(array)
>     || !require_native(array)) SWIG_fail;
>  $1 = 1;
>  for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i);
>  $2 = (DATA_TYPE*) array_data(array);
> }
>
> and try to alter it to allow for a conversion of type:
> (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2)
> where ARRAY1 is size DIM1 * 2 and  ARRAY2 is size DIM1.  Then I can
> %apply this to my function that I mentioned in the last post.
>
> So here are my first two questions:
>
> 1) where is DIM1 used to declare the array size?  I don't see where it
> is used at all, and I need to somewhere multiply it by 2 to declare
> the size of ARRAY1
>
> 2) I am not understanding where $input comes from, so I do not
> understand how to distinguish between ARRAY1 and ARRAY2.  In the
> attempt I have already tried, I think I just use the pointer to ARRAY1
> twice.
>
> If anyone has suggestions on how to solve this problem, thanks!
>
> regards,
> Rich
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090105/a5964e50/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dftmagnitude.zip
Type: application/zip
Size: 10857 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090105/a5964e50/attachment.zip>

From ndbecker2 at gmail.com  Mon Jan  5 08:34:51 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 05 Jan 2009 08:34:51 -0500
Subject: [Numpy-discussion] Zoom fft code
References: <710F2847B0018641891D9A216027636029C39B@ex3.envision.co.il>
	<9457e7c80901050025j520ce5dds26d914d63e332e76@mail.gmail.com>
Message-ID: <gjt29r$5ij$1@ger.gmane.org>

=?ISO-8859-1?Q?St=E9fan_van_der_Walt?= wrote:

> Hi Nadav
> 
> I recall that you posted an implementation yourself a while ago!
> 
> http://www.mail-archive.com/numpy-discussion at scipy.org/msg01812.html
> 
> Regards
> St?fan
> 
> 2009/1/5 Nadav Horesh <nadavh at visionsense.com>:
>>
>>  I am looking for a zoom fft code. I found an old code by Paule Kinzle (a
>>  matlab code with a translation to numarray), but its 2D extension
>>  (czt1.py) looks buggy.
>>
>>  Nadav.

I was not aware that chirp-z transform can be used to efficiently compute DFT over a limited part of the spectrum.  I could use this.  Any references on this technique?



From stefan at sun.ac.za  Mon Jan  5 08:58:11 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 5 Jan 2009 15:58:11 +0200
Subject: [Numpy-discussion] Zoom fft code
In-Reply-To: <gjt29r$5ij$1@ger.gmane.org>
References: <710F2847B0018641891D9A216027636029C39B@ex3.envision.co.il>
	<9457e7c80901050025j520ce5dds26d914d63e332e76@mail.gmail.com>
	<gjt29r$5ij$1@ger.gmane.org>
Message-ID: <9457e7c80901050558v585437bcl5d5e17cb209ebebf@mail.gmail.com>

2009/1/5 Neal Becker <ndbecker2 at gmail.com>:
> I was not aware that chirp-z transform can be used to efficiently compute DFT over a limited part of the spectrum.  I could use this.  Any references on this technique?

The only reference I have is the one mentioned in the source:

Rabiner, L.R., R.W. Schafer and C.M. Rader.
The Chirp z-Transform Algorithm.
IEEE Transactions on Audio and Electroacoustics, AU-17(2):86--92, 1969

The discrete z-transform,

X(z_k) = \sum_{n=0}^{N-1} x_n z^{-n}

is calculated at M points,

z_k = AW^-k, k = 0,1,...,M-1.

You can think of the z_k's as a spiral, where A controls the outside
radius (starting frequency) and W the rate of inward spiralling.

Regards
St?fan


From cimrman3 at ntc.zcu.cz  Mon Jan  5 10:13:48 2009
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Mon, 05 Jan 2009 16:13:48 +0100
Subject: [Numpy-discussion] unique1d and asarray
In-Reply-To: <2983337C-89AE-4FE6-8DAB-A68DA04F0380@gmail.com>
References: <FAB3FF41-98C5-4471-86B0-B876A7672168@gmail.com>	<3d375d730901041347j72f4dba2p36dd3a03ccade2ae@mail.gmail.com>
	<2983337C-89AE-4FE6-8DAB-A68DA04F0380@gmail.com>
Message-ID: <496223AC.1090403@ntc.zcu.cz>

Pierre GM wrote:
> On Jan 4, 2009, at 4:47 PM, Robert Kern wrote:
> 
>> On Sun, Jan 4, 2009 at 15:44, Pierre GM <pgmdevlist at gmail.com> wrote:
>>> If we used np.asanyarray instead, subclasses are recognized properly,
>>> the mask is recognized by argsort and the result correct.
>>> Is there a reason why we use np.asarray instead of np.asanyarray ?
>> Probably not.
> 
> So there wouldn't be any objections to make the switch ? We can wait a  
> couple of days if anybody has a pb with that...

There are probably other functions in arraysetops that could be fixed 
easily to work with masked arrays, feel free to do it if you like. I 
have never worked with the masked arrays, so the np.asarray problem had 
not come to my mind. Also, if you change np.asarray to np.asanyarray, 
add a corresponding test emplying the masked arrays to 
test_arraysetops.py, please.

cheers & thanks,
r.


From nadavh at visionsense.com  Mon Jan  5 10:26:54 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Mon, 5 Jan 2009 17:26:54 +0200
Subject: [Numpy-discussion] Zoom fft code
References: <710F2847B0018641891D9A216027636029C39B@ex3.envision.co.il>
	<9457e7c80901050025j520ce5dds26d914d63e332e76@mail.gmail.com>
Message-ID: <710F2847B0018641891D9A216027636029C39C@ex3.envision.co.il>

Thank you, I lost the code so thank you for finding it. In addition, chirp z transform is broader then zoom fft. There was someone on this list that was interested especially in zoom fft, so I was wondered if there is a code for it. Anyway, I can use my old code again.

   Nadav

  


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? St?fan van der Walt
????: ? 05-?????-09 10:25
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Zoom fft code
 
Hi Nadav

I recall that you posted an implementation yourself a while ago!

http://www.mail-archive.com/numpy-discussion at scipy.org/msg01812.html

Regards
St?fan

2009/1/5 Nadav Horesh <nadavh at visionsense.com>:
>
>  I am looking for a zoom fft code. I found an old code by Paule Kinzle (a matlab code with a translation to numarray), but its 2D extension (czt1.py) looks buggy.
>
>  Nadav.
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3221 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090105/4544cfe9/attachment.bin>

From reakinator at gmail.com  Mon Jan  5 11:06:06 2009
From: reakinator at gmail.com (Rich E)
Date: Mon, 5 Jan 2009 17:06:06 +0100
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
Message-ID: <a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>

Egor,

Thanks for the help.  I think I want to leave the C code as-is
however, as it is perfectly fine there no knowing 'sizeOutMag' because
it can deduce both array sizes from one variable.  There are many
other similar cases in my code (many where the size of the array is
known by a member of a structure passed to the function).

Maybe I should look into using an 'insertion block' of code in the
interface file, instead of trying to typemap the array?  I am thinking
I may just be able to copy the generated code (from SWIG) into my
interface file to do this, but I have not tried it yet.

I will experiment a little and post again.  Thanks and happy holidays!

regards,
Rich

On Mon, Jan 5, 2009 at 10:42 AM, Egor Zindy <ezindy at gmail.com> wrote:
> Hello Rich,
>
> sorry it took so long to answer back, holidays and all :-)
>
> That's exactly the kind of SWIG / numpy.i problems I've been working on over
> the past few months: How to generate an array you don't know the size of
> a-priori, and then handle the memory deallocation seamlessly. In your case,
> you know that the output array will be half the size of the input array, but
> this falls under the more general case of "not knowing the output size
> a-priori".
>
> Have a look at the files attached. I've rewritten your function header as:
> void sms_spectrumMag( int sizeInMag, float *pInRect, int *sizeOutMag, float
> **pOutMag);
>
> Easy to see what the input and output arrays are now. Then my numpy.i
> handles the memory deallocation of the **pOutMag array.
>
> I've actually moved my numpy.i explanations to the scipy/numpy cookbook last
> week :-)
> http://www.scipy.org/Cookbook/SWIG_Memory_Deallocation
>
> Hope it all makes sense. If you have any questions, don't hesitate!
>
>>python test_dftmagnitude.py
> [1, 1, 2, 2]
> [ 1.41421354  2.82842708]
> [1, 1, 2, 2, 3, 3, 4, 4]
> [ 1.41421354  2.82842708  4.2426405   5.65685415]
> [1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
> [ 1.41421354  2.82842708  4.2426405   5.65685415  7.07106781]
>
> Regards,
> Egor
>
> On Wed, Dec 24, 2008 at 1:52 AM, Rich E <reakinator at gmail.com> wrote:
>>
>> Hi list,
>>
>> My question has to do with the Numpy/SWIG typemapping system.
>>
>> I recently got the typemaps in numpy.i to work on most of my C
>> functions that are wrapped using SWIG, if they have arguments of the
>> form (int sizeArray, float *pArray).
>>
>> Now I am trying to figure out how to wrap function that aren't of the
>> form, such as the following function:
>>
>> /*! \brief compute magnitude spectrum of a DFT
>>  *
>>  * \param sizeMag              size of output Magnitude (half of input
>> real FFT)
>>  * \param pFReal               pointer to input FFT real array
>> (real/imag floats)
>>  * \param pFMAg        pointer to float array of magnitude spectrum
>>  */
>> void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag)
>> {
>>       int i, it2;
>>       float fReal, fImag;
>>
>>       for (i=0; i<sizeMag; i++)
>>       {
>>               it2 = i << 1;
>>               fReal = pInRect[it2];
>>               fImag = pInRect[it2+1];
>>               pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag);
>>       }
>> }
>>
>> There are two arrays, one is half the size of the other.  But, SWIG
>> doesn't know this, according to the type map it will think *pInRect is
>> of size sizeMag and will not know anything about *pOutMag.
>>
>> Ideally in python, I would like to call the function as
>> sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of
>> nArray2, and nArray2 is of size sizeMag.
>>
>> I think in order to do this (although if someone has a better
>> suggestion, I am open to it), I will have to modify the typemap in
>> order to tell SWIG how to call the C function properly.  I do not want
>> to have to edit the wrapped C file every time it is regenerated from
>> the interface file.
>>
>>
>> Here is a start I made with the existing typemap code in numpy.i (not
>> working):
>>
>> /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>>  */
>> %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY,
>>          fragment="NumPy_Macros")
>>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>> {
>>  $1 = is_array($input) && PyArray_EquivTypenums(array_type($input),
>>                                                DATA_TYPECODE);
>> }
>> %typemap(in,
>>        fragment="NumPy_Fragments")
>>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>>  (PyArrayObject* array=NULL, int i=0)
>> {
>>  array = obj_to_array_no_conversion($input, DATA_TYPECODE);
>>  if (!array || !require_dimensions(array,1) || !require_contiguous(array)
>>     || !require_native(array)) SWIG_fail;
>>  $1 = 1;
>>  for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i);
>>  $2 = (DATA_TYPE*) array_data(array);
>> }
>>
>> and try to alter it to allow for a conversion of type:
>> (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2)
>> where ARRAY1 is size DIM1 * 2 and  ARRAY2 is size DIM1.  Then I can
>> %apply this to my function that I mentioned in the last post.
>>
>> So here are my first two questions:
>>
>> 1) where is DIM1 used to declare the array size?  I don't see where it
>> is used at all, and I need to somewhere multiply it by 2 to declare
>> the size of ARRAY1
>>
>> 2) I am not understanding where $input comes from, so I do not
>> understand how to distinguish between ARRAY1 and ARRAY2.  In the
>> attempt I have already tried, I think I just use the pointer to ARRAY1
>> twice.
>>
>> If anyone has suggestions on how to solve this problem, thanks!
>>
>> regards,
>> Rich
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From garry.willgoose at newcastle.edu.au  Mon Jan  5 19:48:57 2009
From: garry.willgoose at newcastle.edu.au (Garry Willgoose)
Date: Tue, 6 Jan 2009 11:48:57 +1100
Subject: [Numpy-discussion] when will osx linker option -bundle be reflected
	in distutils
Message-ID: <3993B772-CE2D-436E-AA88-336C8ADBBFCB@newcastle.edu.au>

> > I was just wondering what plans there were to reflect the different
> > linker options (i.e. -bundle instead of -shared) that are required  
> on
> > OSX in the fcompiler files within distutils. While its a minor thing
> > it always catches the users of my software when they either install
> > fresh or update numpy ... and sometimes on a bad day it even catches
> > me ;-)
>
> I'm sorry; I don't follow. What problems are you having? --
> -- Robert Kern
>
-----------------------------------------------

OK for example the distribution g95.py in distutils/fcompiler has the  
following code

        executables = {
            'version_cmd'  : ["g95", "--version"],
            'compiler_f77' : ["g95", "-ffixed-form"],
            'compiler_fix' : ["g95", "-ffixed-form"],
            'compiler_f90' : ["g95"],
            'linker_so'    : ["g95","-shared"],
            'archiver'     : ["ar", "-cr"],
            'ranlib'       : ["ranlib"]
            }

For osx you need to modify it to

      executables = {
            'version_cmd'  : ["g95", "--version"],
            'compiler_f77' : ["g95", "-ffixed-form"],
            'compiler_fix' : ["g95", "-ffixed-form"],
            'compiler_f90' : ["g95"],
            'linker_so'    : ["g95","-shared"],
            'archiver'     : ["ar", "-cr"],
            'ranlib'       : ["ranlib"]
            }
    import sys
    if sys.platform.lower() == 'darwin':
        executables[linker_so'] = ["g95","-Wall -bundle"]

The 'shared' option is not implemented in the osx linker. Not sure  
what the underlying difference between 'shared' and 'bundle' is but  
this substitution is necessary and this has been working for me for  
the last year or so. You also need the -Wall but for reasons that  
completely escape me.

The same goes for gfortran and intel (both of which I use) and I  
assume the other compilers that are available for OSX.


====================================================================
Prof Garry Willgoose,
Australian Professorial Fellow in Environmental Engineering,
Director, Centre for Climate Impact Management (C2IM),
School of Engineering, The University of Newcastle,
Callaghan, 2308
Australia.

Centre webpage: www.c2im.org.au

Phone: (International) +61 2 4921 6050 (Tues-Fri AM); +61 2 6545 9574  
(Fri PM-Mon)
FAX: (International) +61 2 4921 6991 (Uni); +61 2 6545 9574 (personal  
and Telluric)
Env. Engg. Secretary: (International) +61 2 4921 6042

email:  garry.willgoose at newcastle.edu.au; g.willgoose at telluricresearch.com
email-for-life: garry.willgoose at alum.mit.edu
personal webpage: www.telluricresearch.com/garry
====================================================================
"Do not go where the path may lead, go instead where there is no path  
and leave a trail"
                           Ralph Waldo Emerson
====================================================================







From robert.kern at gmail.com  Mon Jan  5 20:30:12 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 5 Jan 2009 19:30:12 -0600
Subject: [Numpy-discussion] when will osx linker option -bundle be
	reflected in distutils
In-Reply-To: <3993B772-CE2D-436E-AA88-336C8ADBBFCB@newcastle.edu.au>
References: <3993B772-CE2D-436E-AA88-336C8ADBBFCB@newcastle.edu.au>
Message-ID: <3d375d730901051730o28c509d5g7bcdb8d8ad712815@mail.gmail.com>

On Mon, Jan 5, 2009 at 18:48, Garry Willgoose
<garry.willgoose at newcastle.edu.au> wrote:
>> > I was just wondering what plans there were to reflect the different
>> > linker options (i.e. -bundle instead of -shared) that are required
>> on
>> > OSX in the fcompiler files within distutils. While its a minor thing
>> > it always catches the users of my software when they either install
>> > fresh or update numpy ... and sometimes on a bad day it even catches
>> > me ;-)
>>
>> I'm sorry; I don't follow. What problems are you having? --
>> -- Robert Kern
>>
> -----------------------------------------------
>
> OK for example the distribution g95.py in distutils/fcompiler has the
> following code
>
>        executables = {
>            'version_cmd'  : ["g95", "--version"],
>            'compiler_f77' : ["g95", "-ffixed-form"],
>            'compiler_fix' : ["g95", "-ffixed-form"],
>            'compiler_f90' : ["g95"],
>            'linker_so'    : ["g95","-shared"],
>            'archiver'     : ["ar", "-cr"],
>            'ranlib'       : ["ranlib"]
>            }
>
> For osx you need to modify it to
>
>      executables = {
>            'version_cmd'  : ["g95", "--version"],
>            'compiler_f77' : ["g95", "-ffixed-form"],
>            'compiler_fix' : ["g95", "-ffixed-form"],
>            'compiler_f90' : ["g95"],
>            'linker_so'    : ["g95","-shared"],
>            'archiver'     : ["ar", "-cr"],
>            'ranlib'       : ["ranlib"]
>            }
>    import sys
>    if sys.platform.lower() == 'darwin':
>        executables[linker_so'] = ["g95","-Wall -bundle"]
>
> The 'shared' option is not implemented in the osx linker. Not sure
> what the underlying difference between 'shared' and 'bundle' is but
> this substitution is necessary and this has been working for me for
> the last year or so. You also need the -Wall but for reasons that
> completely escape me.

-Wall absolutely should not affect anything except adding warning
messages. I suspect something else is getting modified when you do
that.

> The same goes for gfortran and intel (both of which I use) and I
> assume the other compilers that are available for OSX.

I've been building scipy for years with gfortran and an unmodified
numpy on OS X. The correct switches are added in the
get_flags_linker_so() method:

    def get_flags_linker_so(self):
        opt = self.linker_so[1:]
        if sys.platform=='darwin':
            # MACOSX_DEPLOYMENT_TARGET must be at least 10.3. This is
            # a reasonable default value even when building on 10.4 when using
            # the official Python distribution and those derived from it (when
            # not broken).
            target = os.environ.get('MACOSX_DEPLOYMENT_TARGET', None)
            if target is None or target == '':
                target = '10.3'
            major, minor = target.split('.')
            if int(minor) < 3:
                minor = '3'
                warnings.warn('Environment variable '
                    'MACOSX_DEPLOYMENT_TARGET reset to %s.%s' % (major, minor))
            os.environ['MACOSX_DEPLOYMENT_TARGET'] = '%s.%s' % (major,
                minor)

            opt.extend(['-undefined', 'dynamic_lookup', '-bundle'])
        else:
            opt.append("-shared")
        if sys.platform.startswith('sunos'):
            # SunOS often has dynamically loaded symbols defined in the
            # static library libg2c.a  The linker doesn't like this.  To
            # ignore the problem, use the -mimpure-text flag.  It isn't
            # the safest thing, but seems to work. 'man gcc' says:
            # ".. Instead of using -mimpure-text, you should compile all
            #  source code with -fpic or -fPIC."
            opt.append('-mimpure-text')
        return opt

If this is not working for you, please show me the error messages you get.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From fonnesbeck at maths.otago.ac.nz  Mon Jan  5 21:41:57 2009
From: fonnesbeck at maths.otago.ac.nz (Christopher Fonnesbeck)
Date: Tue, 6 Jan 2009 15:41:57 +1300
Subject: [Numpy-discussion] ANN: PyMC 2.0
Message-ID: <7DB2E7D8-A9CF-4F3F-9979-71BC5F26DFDD@maths.otago.ac.nz>

Numpy list members,

It gives me great pleasure to be able to announce the long-awaited  
release of PyMC 2.0. Platform-specific installers have been uploaded  
to the Google Code page (Mac OSX) and the Python Package Index (all  
other platforms), along with the new user's guide (http://pymc.googlecode.com/files/UserGuide2.0.pdf 
).

PyMC is a python module that implements Bayesian statistical models  
and ?tting algorithms, including Markov chain Monte Carlo. Its  
?exibility makes it applicable to a large suite of problems as well  
as easily extensible. Along with core sampling functionality, PyMC  
includes methods for summarizing output, plotting, goodness-of-?t and  
convergence diagnostics.

PyMC 2.0 is a quantum leap from the 1.3 release. It includes a  
completely revised object model and syntax, more efficient log- 
probability computation, a variety of specialised MCMC algorithms, and  
an expanded set of optimised probability distributions. As a result,  
models built for previous versions of PyMC will not run under version  
2.0.

I would like to particularly thank Anand Patil and David Huard, who  
have done most of the work on this version, and to all the users who  
have sent questions, comments and bug reports over the past year or  
two. Please keep the feedback coming!

Please report any problems with the release to the issues page (http://code.google.com/p/pymc/issues/list 
).

Python Package Index: http://pypi.python.org/pypi/pymc/
Google Code: http://pymc.googelcode.com
Mailing List: http://groups.google.com/group/pymc

Happy new year,
Chris
--
Christopher J. Fonnesbeck
Department of Mathematics and Statistics
University of Otago, PO Box 56
Dunedin, New Zealand



From dmacks at netspace.org  Mon Jan  5 23:11:30 2009
From: dmacks at netspace.org (Daniel Macks)
Date: Mon, 5 Jan 2009 23:11:30 -0500
Subject: [Numpy-discussion] when will osx linker option -bundle be
	reflected in distutils
In-Reply-To: <3993B772-CE2D-436E-AA88-336C8ADBBFCB@newcastle.edu.au>
References: <3993B772-CE2D-436E-AA88-336C8ADBBFCB@newcastle.edu.au>
Message-ID: <20090106041130.GA21304@happy.netspace.org>

On Tue, Jan 06, 2009 at 11:48:57AM +1100, Garry Willgoose wrote:
> The 'shared' option is not implemented in the osx linker. Not sure  
> what the underlying difference between 'shared' and 'bundle' is

To answer this narrow part of the question, -shared is the way to
build shared libraries on linux (I think it's part of the standard GNU
ld and/or ELF binary format), and is how one builds all sorts of .so.
On OS X, there is a difference between a "dynamic library" (one that
is linked later via "-lFOO" flags, standard extension .dylib) and a
"loadable module" (one that is loaded at runtime via dlopen() or
similar methods, often extension .so). Linux doesn't have as sharp a
distinction. OS X linker uses different flags to specify which one to
build (-dynamiclib and -bundle, respectively).

dan

-- 
Daniel Macks
dmacks at netspace.org
http://www.netspace.org/~dmacks



From stefan at sun.ac.za  Tue Jan  6 04:15:45 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 6 Jan 2009 11:15:45 +0200
Subject: [Numpy-discussion] Handling methods of object-arrays
Message-ID: <9457e7c80901060115wb142238i1fbcca5ebae02299@mail.gmail.com>

Hi all,

What is the exact protocol for evaluating functions like "real" and
"imag" on object arrays?

For example, I'm looking at

x  = np.array([np.array(3+1j), np.array(4+1j)], dtype=object)

For which both

In [4]: x.real
Out[4]: array([(3+1j), (4+1j)], dtype=object)

and

In [6]: np.real(x)
Out[6]: array([(3+1j), (4+1j)], dtype=object)

does nothing, so that I have to do

In [8]: [np.real(e) for e in x]
Out[8]: [array(3.0), array(4.0)]

or [e.real for e in x].

Would it make sense make np.real aware of the above scenario?

Regards
St?fan


From robert.kern at gmail.com  Tue Jan  6 04:19:10 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 6 Jan 2009 03:19:10 -0600
Subject: [Numpy-discussion] Handling methods of object-arrays
In-Reply-To: <9457e7c80901060115wb142238i1fbcca5ebae02299@mail.gmail.com>
References: <9457e7c80901060115wb142238i1fbcca5ebae02299@mail.gmail.com>
Message-ID: <3d375d730901060119k6e124d15lc2d66e36102fbabc@mail.gmail.com>

On Tue, Jan 6, 2009 at 03:15, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi all,
>
> What is the exact protocol for evaluating functions like "real" and
> "imag" on object arrays?
>
> For example, I'm looking at
>
> x  = np.array([np.array(3+1j), np.array(4+1j)], dtype=object)
>
> For which both
>
> In [4]: x.real
> Out[4]: array([(3+1j), (4+1j)], dtype=object)
>
> and
>
> In [6]: np.real(x)
> Out[6]: array([(3+1j), (4+1j)], dtype=object)
>
> does nothing, so that I have to do
>
> In [8]: [np.real(e) for e in x]
> Out[8]: [array(3.0), array(4.0)]
>
> or [e.real for e in x].
>
> Would it make sense make np.real aware of the above scenario?

Methods like ndarray.sin() check for e.sin(), so a case could
certainly be made that ndarray.real should check for e.real .

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From pommereau at univ-paris12.fr  Tue Jan  6 04:35:01 2009
From: pommereau at univ-paris12.fr (Franck Pommereau)
Date: Tue, 06 Jan 2009 10:35:01 +0100
Subject: [Numpy-discussion] [Newbie] Fast plotting
Message-ID: <496325C5.4000505@univ-paris12.fr>

Hi all, and happy new year!

I'm new to NumPy and searching a way to compute from a set of points
(x,y) the mean value of y values associated to each distinct x value.
Each point corresponds to a measure in a benchmark (x = parameter,  y =
computation time) and I'd like to plot the graph of mean computation
time wrt parameter values. (I know how to plot, but not how to compute
mean values.)

My points are stored as two arrays X, Y (same size).
In pure Python, I'd do as follows:

s = {} # sum of y values for each distinct x (as keys)
n = {} # number of summed values (same keys)
for x, y in zip(X, Y) :
    s[x] = s.get(x, 0.0) + y
    n[x] = n.get(x, 0) + 1
new_x = array(list(sorted(s)))
new_y = array([s[x]/n[x] for x in sorted(s)])

Unfortunately, this code is much too slow because my arrays have
millions of elements. But I'm pretty sure that NumPy offers a way to
handle this more elegantly and much faster.

As a bonus, I'd be happy if the solution would allow me to compute also
standard deviation, min, max, etc.

Thanks in advance for any help!
Franck


From stefan at sun.ac.za  Tue Jan  6 05:14:53 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 6 Jan 2009 12:14:53 +0200
Subject: [Numpy-discussion] Old-style classes in tests
In-Reply-To: <492ACBCB.4060204@resolversystems.com>
References: <492ACBCB.4060204@resolversystems.com>
Message-ID: <9457e7c80901060214o68bf8cdeq123f234ed6d2dfce@mail.gmail.com>

Hi Tom

2008/11/24 Tom Wright <tom.wright at resolversystems.com>:
> I am currently working on the Ironclad project porting numpy to Ironpython.
>
> It would be quite useful for me if HermitianTestCase in test_linalg.py
> was a new style-class instead of an old-style class - since Ironpython
> has a bug where dir operations do not work for classes inheriting from
> both old- and new- style classes and I'd very much prefer not to patch
> my version of numpy.

These changes have been made in r6297.  Let me know if you find any
other occurrences.

Regards
St?fan


From faltet at pytables.org  Tue Jan  6 06:56:44 2009
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 6 Jan 2009 12:56:44 +0100
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <496325C5.4000505@univ-paris12.fr>
References: <496325C5.4000505@univ-paris12.fr>
Message-ID: <200901061256.44641.faltet@pytables.org>

A Tuesday 06 January 2009, Franck Pommereau escrigu?:
> Hi all, and happy new year!
>
> I'm new to NumPy and searching a way to compute from a set of points
> (x,y) the mean value of y values associated to each distinct x value.
> Each point corresponds to a measure in a benchmark (x = parameter,  y
> = computation time) and I'd like to plot the graph of mean
> computation time wrt parameter values. (I know how to plot, but not
> how to compute mean values.)
>
> My points are stored as two arrays X, Y (same size).
> In pure Python, I'd do as follows:
>
> s = {} # sum of y values for each distinct x (as keys)
> n = {} # number of summed values (same keys)
> for x, y in zip(X, Y) :
>     s[x] = s.get(x, 0.0) + y
>     n[x] = n.get(x, 0) + 1
> new_x = array(list(sorted(s)))
> new_y = array([s[x]/n[x] for x in sorted(s)])
>
> Unfortunately, this code is much too slow because my arrays have
> millions of elements. But I'm pretty sure that NumPy offers a way to
> handle this more elegantly and much faster.
>
> As a bonus, I'd be happy if the solution would allow me to compute
> also standard deviation, min, max, etc.

The next would do the trick:

In [92]: x = np.random.randint(100,size=100)

In [93]: y = np.random.rand(100)

In [94]: u = np.unique(x)

In [95]: means = [ y[x == i].mean() for i in u ]

In [96]: stds = [ y[x == i].std() for i in u ]

In [97]: maxs = [ y[x == i].max() for i in u ]

In [98]: mins = [ y[x == i].min() for i in u ]

and your wanted data will be in means, stds, maxs and mins lists.  This 
approach has the drawback that you have to process the array each time 
that you want to extract the desired info.  If what you want is to 
always retrieve the same set of statistics, you can do this in one 
single loop:

In [99]: means, std, maxs, mins = [], [], [], []

In [100]: for i in u:
    g = y[x == i]
    means.append(g.mean())
    stds.append(g.std())
    maxs.append(g.max())
    mins.append(g.min())
   .....:

which has the same effect than above, but is much faster.

Hope that helps,

-- 
Francesc Alted


From boogaloojb at yahoo.fr  Tue Jan  6 08:38:27 2009
From: boogaloojb at yahoo.fr (Jean-Baptiste Rudant)
Date: Tue, 6 Jan 2009 13:38:27 +0000 (GMT)
Subject: [Numpy-discussion] Re :  [Newbie] Fast plotting
Message-ID: <970036.60257.qm@web28512.mail.ukl.yahoo.com>

Hello,

I'm not an expert. Something exists in matplotlib, but it's not very efficient.

import matplotlib.mlab
import numpy
N = 1000
X  = numpy.random.randint(0, 10, N)
Y = numpy.random.random(N)
recXY = numpy.rec.fromarrays((X, Y), names='x, y')
summary = matplotlib..mlab.rec_groupby(recXY, ('x',), (('y', numpy.mean, 'y_avg'),))

Jean-Baptiste Rudant




________________________________
De : Franck Pommereau <pommereau at univ-paris12.fr>
? : Discussion of Numerical Python <numpy-discussion at scipy.org>
Envoy? le : Mardi, 6 Janvier 2009, 10h35mn 01s
Objet : [Numpy-discussion] [Newbie] Fast plotting

Hi all, and happy new year!

I'm new to NumPy and searching a way to compute from a set of points
(x,y) the mean value of y values associated to each distinct x value.
Each point corresponds to a measure in a benchmark (x = parameter,  y =
computation time) and I'd like to plot the graph of mean computation
time wrt parameter values. (I know how to plot, but not how to compute
mean values.)

My points are stored as two arrays X, Y (same size).
In pure Python, I'd do as follows:

s = {} # sum of y values for each distinct x (as keys)
n = {} # number of summed values (same keys)
for x, y in zip(X, Y) :
    s[x] = s.get(x, 0.0) + y
    n[x] = n.get(x, 0) + 1
new_x = array(list(sorted(s)))
new_y = array([s[x]/n[x] for x in sorted(s)])

Unfortunately, this code is much too slow because my arrays have
millions of elements. But I'm pretty sure that NumPy offers a way to
handle this more elegantly and much faster.

As a bonus, I'd be happy if the solution would allow me to compute also
standard deviation, min, max, etc.

Thanks in advance for any help!
Franck
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090106/a44d4e07/attachment.html>

From jdh2358 at gmail.com  Tue Jan  6 09:34:02 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Tue, 6 Jan 2009 08:34:02 -0600
Subject: [Numpy-discussion] Re : [Newbie] Fast plotting
In-Reply-To: <970036.60257.qm@web28512.mail.ukl.yahoo.com>
References: <970036.60257.qm@web28512.mail.ukl.yahoo.com>
Message-ID: <88e473830901060634t4e60c1eap78d33b0219cfca7e@mail.gmail.com>

On Tue, Jan 6, 2009 at 7:38 AM, Jean-Baptiste Rudant
<boogaloojb at yahoo.fr> wrote:
> Hello,
> I'm not an expert. Something exists in matplotlib, but it's not very
> efficient.
> import matplotlib.mlab
> import numpy
> N = 1000
> X  = numpy.random.randint(0, 10, N)
> Y = numpy.random.random(N)
> recXY = numpy.rec.fromarrays((X, Y), names='x, y')
> summary = matplotlib.mlab.rec_groupby(recXY, ('x',), (('y', numpy.mean,
> 'y_avg'),))

And you can use rec2txt for pretty printing in the shell:

In [103]: print matplotlib.mlab.rec2txt(summary)
   x   y_avg
   0   0.506
   1   0.531
   2   0.491
   3   0.482
   4   0.511
   5   0.507
   6   0.543
   7   0.525
   8   0.512
   9   0.472


> Jean-Baptiste Rudant
>
> ________________________________
> De : Franck Pommereau <pommereau at univ-paris12.fr>
> ? : Discussion of Numerical Python <numpy-discussion at scipy.org>
> Envoy? le : Mardi, 6 Janvier 2009, 10h35mn 01s
> Objet : [Numpy-discussion] [Newbie] Fast plotting
>
> Hi all, and happy new year!
>
> I'm new to NumPy and searching a way to compute from a set of points
> (x,y) the mean value of y values associated to each distinct x value.
> Each point corresponds to a measure in a benchmark (x = parameter,  y =
> computation time) and I'd like to plot the graph of mean computation
> time wrt parameter values. (I know how to plot, but not how to compute
> mean values.)
>
> My points are stored as two arrays X, Y (same size).
> In pure Python, I'd do as follows:
>
> s = {} # sum of y values for each distinct x (as keys)
> n = {} # number of summed values (same keys)
> for x, y in zip(X, Y) :
>     s[x] = s.get(x, 0.0) + y
>     n[x] = n.get(x, 0) + 1
> new_x = array(list(sorted(s)))
> new_y = array([s[x]/n[x] for x in sorted(s)])
>
> Unfortunately, this code is much too slow because my arrays have
> millions of elements. But I'm pretty sure that NumPy offers a way to
> handle this more elegantly and much faster.
>
> As a bonus, I'd be happy if the solution would allow me to compute also
> standard deviation, min, max, etc.
>
> Thanks in advance for any help!
> Franck
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From bsouthey at gmail.com  Tue Jan  6 09:44:42 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 06 Jan 2009 08:44:42 -0600
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <200901061256.44641.faltet@pytables.org>
References: <496325C5.4000505@univ-paris12.fr>
	<200901061256.44641.faltet@pytables.org>
Message-ID: <49636E5A.2070301@gmail.com>

Francesc Alted wrote:
> A Tuesday 06 January 2009, Franck Pommereau escrigu?:
>   
>> Hi all, and happy new year!
>>
>> I'm new to NumPy and searching a way to compute from a set of points
>> (x,y) the mean value of y values associated to each distinct x value.
>> Each point corresponds to a measure in a benchmark (x = parameter,  y
>> = computation time) and I'd like to plot the graph of mean
>> computation time wrt parameter values. (I know how to plot, but not
>> how to compute mean values.)
>>
>> My points are stored as two arrays X, Y (same size).
>> In pure Python, I'd do as follows:
>>
>> s = {} # sum of y values for each distinct x (as keys)
>> n = {} # number of summed values (same keys)
>> for x, y in zip(X, Y) :
>>     s[x] = s.get(x, 0.0) + y
>>     n[x] = n.get(x, 0) + 1
>> new_x = array(list(sorted(s)))
>> new_y = array([s[x]/n[x] for x in sorted(s)])
>>
>> Unfortunately, this code is much too slow because my arrays have
>> millions of elements. But I'm pretty sure that NumPy offers a way to
>> handle this more elegantly and much faster.
>>
>> As a bonus, I'd be happy if the solution would allow me to compute
>> also standard deviation, min, max, etc.
>>     
>
> The next would do the trick:
>
> In [92]: x = np.random.randint(100,size=100)
>
> In [93]: y = np.random.rand(100)
>
> In [94]: u = np.unique(x)
>
> In [95]: means = [ y[x == i].mean() for i in u ]
>
> In [96]: stds = [ y[x == i].std() for i in u ]
>
> In [97]: maxs = [ y[x == i].max() for i in u ]
>
> In [98]: mins = [ y[x == i].min() for i in u ]
>
> and your wanted data will be in means, stds, maxs and mins lists.  This 
> approach has the drawback that you have to process the array each time 
> that you want to extract the desired info.  If what you want is to 
> always retrieve the same set of statistics, you can do this in one 
> single loop:
>
> In [99]: means, std, maxs, mins = [], [], [], []
>
> In [100]: for i in u:
>     g = y[x == i]
>     means.append(g.mean())
>     stds.append(g.std())
>     maxs.append(g.max())
>     mins.append(g.min())
>    .....:
>
> which has the same effect than above, but is much faster.
>
> Hope that helps,
>
>   
If you use Knuth's one pass approach 
(http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#III._On-line_algorithm) 
you can write a function to get the min, max, mean and variance/standard 
deviation in a single pass through the array rather than one pass for 
each. I do not know if this will provide any advantage as that will 
probably depend on the size of the arrays.

Also, please use the highest precision possible (ie float128) for your 
arrays to minimize numerical error due to the size of your arrays.

Bruce


From ezindy at gmail.com  Tue Jan  6 09:47:38 2009
From: ezindy at gmail.com (Egor Zindy)
Date: Tue, 6 Jan 2009 23:47:38 +0900
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
Message-ID: <c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>

Hello again,

I really don't know what came over me when I changed your function
prototype, that wasn't a very thoughtful thing to do!

Maybe I should look into using an 'insertion block' of code in the
> interface file, instead of trying to typemap the array?
>

Insertion blocks... is that %inline code? In which case, yes! Have a look, I
attached a new version that uses some %inline directives in the
dftmagnitude.i file.

Basically, you can inline a new function with an easier prototype to wrap.
The function allocates memory and calls your sms_spectrumMag() function.

my inline function: void my_spectrumMag( int sizeInMag, float *pInRect, int
*sizeOutMag, float **pOutMag)

there's also a %rename directive: %rename (spectrumMag) my_spectrumMag;

I had a go at defining some exceptions too (no memory and odd number of
indexes), but I'm not sure errno is the easiest way to go about it...

Hope this helps!

... and the python test output:

~>python test_dftmagnitude.py
array: [1, 1, 2, 2]
result: [ 1.41421354  2.82842708]

array: [1, 1, 2, 2, 3, 3, 4, 4]
result: [ 1.41421354  2.82842708  4.2426405   5.65685415]

array: [1, 1, 2]
result:
Traceback (most recent call last):
  File "test_dftmagnitude.py", line 15, in <module>
    print "result:",dftmagnitude.spectrumMag(a)
IndexError: Odd number of elements in input array: 3

~>

Regards,
Egor


On Tue, Jan 6, 2009 at 1:06 AM, Rich E <reakinator at gmail.com> wrote:

> Egor,
>
> Thanks for the help.  I think I want to leave the C code as-is
> however, as it is perfectly fine there no knowing 'sizeOutMag' because
> it can deduce both array sizes from one variable.  There are many
> other similar cases in my code (many where the size of the array is
> known by a member of a structure passed to the function).
>
> Maybe I should look into using an 'insertion block' of code in the
> interface file, instead of trying to typemap the array?  I am thinking
> I may just be able to copy the generated code (from SWIG) into my
> interface file to do this, but I have not tried it yet.
>
> I will experiment a little and post again.  Thanks and happy holidays!
>
> regards,
> Rich
>
> On Mon, Jan 5, 2009 at 10:42 AM, Egor Zindy <ezindy at gmail.com> wrote:
> > Hello Rich,
> >
> > sorry it took so long to answer back, holidays and all :-)
> >
> > That's exactly the kind of SWIG / numpy.i problems I've been working on
> over
> > the past few months: How to generate an array you don't know the size of
> > a-priori, and then handle the memory deallocation seamlessly. In your
> case,
> > you know that the output array will be half the size of the input array,
> but
> > this falls under the more general case of "not knowing the output size
> > a-priori".
> >
> > Have a look at the files attached. I've rewritten your function header
> as:
> > void sms_spectrumMag( int sizeInMag, float *pInRect, int *sizeOutMag,
> float
> > **pOutMag);
> >
> > Easy to see what the input and output arrays are now. Then my numpy.i
> > handles the memory deallocation of the **pOutMag array.
> >
> > I've actually moved my numpy.i explanations to the scipy/numpy cookbook
> last
> > week :-)
> > http://www.scipy.org/Cookbook/SWIG_Memory_Deallocation
> >
> > Hope it all makes sense. If you have any questions, don't hesitate!
> >
> >>python test_dftmagnitude.py
> > [1, 1, 2, 2]
> > [ 1.41421354  2.82842708]
> > [1, 1, 2, 2, 3, 3, 4, 4]
> > [ 1.41421354  2.82842708  4.2426405   5.65685415]
> > [1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
> > [ 1.41421354  2.82842708  4.2426405   5.65685415  7.07106781]
> >
> > Regards,
> > Egor
> >
> > On Wed, Dec 24, 2008 at 1:52 AM, Rich E <reakinator at gmail.com> wrote:
> >>
> >> Hi list,
> >>
> >> My question has to do with the Numpy/SWIG typemapping system.
> >>
> >> I recently got the typemaps in numpy.i to work on most of my C
> >> functions that are wrapped using SWIG, if they have arguments of the
> >> form (int sizeArray, float *pArray).
> >>
> >> Now I am trying to figure out how to wrap function that aren't of the
> >> form, such as the following function:
> >>
> >> /*! \brief compute magnitude spectrum of a DFT
> >>  *
> >>  * \param sizeMag              size of output Magnitude (half of input
> >> real FFT)
> >>  * \param pFReal               pointer to input FFT real array
> >> (real/imag floats)
> >>  * \param pFMAg        pointer to float array of magnitude spectrum
> >>  */
> >> void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag)
> >> {
> >>       int i, it2;
> >>       float fReal, fImag;
> >>
> >>       for (i=0; i<sizeMag; i++)
> >>       {
> >>               it2 = i << 1;
> >>               fReal = pInRect[it2];
> >>               fImag = pInRect[it2+1];
> >>               pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag);
> >>       }
> >> }
> >>
> >> There are two arrays, one is half the size of the other.  But, SWIG
> >> doesn't know this, according to the type map it will think *pInRect is
> >> of size sizeMag and will not know anything about *pOutMag.
> >>
> >> Ideally in python, I would like to call the function as
> >> sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of
> >> nArray2, and nArray2 is of size sizeMag.
> >>
> >> I think in order to do this (although if someone has a better
> >> suggestion, I am open to it), I will have to modify the typemap in
> >> order to tell SWIG how to call the C function properly.  I do not want
> >> to have to edit the wrapped C file every time it is regenerated from
> >> the interface file.
> >>
> >>
> >> Here is a start I made with the existing typemap code in numpy.i (not
> >> working):
> >>
> >> /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
> >>  */
> >> %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY,
> >>          fragment="NumPy_Macros")
> >>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
> >> {
> >>  $1 = is_array($input) && PyArray_EquivTypenums(array_type($input),
> >>                                                DATA_TYPECODE);
> >> }
> >> %typemap(in,
> >>        fragment="NumPy_Fragments")
> >>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
> >>  (PyArrayObject* array=NULL, int i=0)
> >> {
> >>  array = obj_to_array_no_conversion($input, DATA_TYPECODE);
> >>  if (!array || !require_dimensions(array,1) ||
> !require_contiguous(array)
> >>     || !require_native(array)) SWIG_fail;
> >>  $1 = 1;
> >>  for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i);
> >>  $2 = (DATA_TYPE*) array_data(array);
> >> }
> >>
> >> and try to alter it to allow for a conversion of type:
> >> (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2)
> >> where ARRAY1 is size DIM1 * 2 and  ARRAY2 is size DIM1.  Then I can
> >> %apply this to my function that I mentioned in the last post.
> >>
> >> So here are my first two questions:
> >>
> >> 1) where is DIM1 used to declare the array size?  I don't see where it
> >> is used at all, and I need to somewhere multiply it by 2 to declare
> >> the size of ARRAY1
> >>
> >> 2) I am not understanding where $input comes from, so I do not
> >> understand how to distinguish between ARRAY1 and ARRAY2.  In the
> >> attempt I have already tried, I think I just use the pointer to ARRAY1
> >> twice.
> >>
> >> If anyone has suggestions on how to solve this problem, thanks!
> >>
> >> regards,
> >> Rich
> >> _______________________________________________
> >> Numpy-discussion mailing list
> >> Numpy-discussion at scipy.org
> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090106/1208a4a1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dftmagnitude2.zip
Type: application/zip
Size: 10880 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090106/1208a4a1/attachment.zip>

From sebastian at sipsolutions.net  Tue Jan  6 09:45:31 2009
From: sebastian at sipsolutions.net (Sebastian Stephan Berg)
Date: Tue, 06 Jan 2009 15:45:31 +0100
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <496325C5.4000505@univ-paris12.fr>
References: <496325C5.4000505@univ-paris12.fr>
Message-ID: <1231253131.6741.50.camel@sebook>

Hello,

Just thinking. If the parameters are limited, you may be able to use the
histogram feature? Doing one histogram with Y as weights, then one
without weights and calculating the mean from this yourself should be
pretty speedy I imagine. Other then that maybe sorting the whole thing
and then doing some searchsorted and side='right' and working on those
slices maybe. I mean something like this:

def spam(x, y, work_on_copy=False):
    """Take the arrays x and y and return
    unique_x_values, means, stds, maxs, mins
    as lists. means, stds, maxs and mins are those
    of the corresponding y values.
    If work_on_copy is true, x and y are copied to ensure
    that they are not sorted in place.
    """
    
    u, means, stds, maxs, mins = [], [], [], [], []
    
    s = x.argsort()
    if work_on_copy:
        x = x[s]
        y = y[s]    
    else:
        x[:] = x[s]
        y[:] = y[s]

    start = 0
    value = x[0]
    while True:
        next = x.searchsorted(value, side='right')
        u.append(value)
        means.append(y[start:next].mean())
        stds.append(y[start:next].std())
        maxs.append(y[start:next].max())
        mins.append(y[start:next].min())
        if next == len(x):
            break    
        value = x[next]    
        start = next
    
    return u, means, stds, maxs, mins

This is of course basically the same as what Francesc suggested, but a
quick test shows that it seems to scale better. I didn't try the speed
of histogram.

Sebastian


On Tue, 2009-01-06 at 10:35 +0100, Franck Pommereau wrote:
> Hi all, and happy new year!
> 
> I'm new to NumPy and searching a way to compute from a set of points
> (x,y) the mean value of y values associated to each distinct x value.
> Each point corresponds to a measure in a benchmark (x = parameter,  y =
> computation time) and I'd like to plot the graph of mean computation
> time wrt parameter values. (I know how to plot, but not how to compute
> mean values.)
> 
> My points are stored as two arrays X, Y (same size).
> In pure Python, I'd do as follows:
> 
> s = {} # sum of y values for each distinct x (as keys)
> n = {} # number of summed values (same keys)
> for x, y in zip(X, Y) :
>     s[x] = s.get(x, 0.0) + y
>     n[x] = n.get(x, 0) + 1
> new_x = array(list(sorted(s)))
> new_y = array([s[x]/n[x] for x in sorted(s)])
> 
> Unfortunately, this code is much too slow because my arrays have
> millions of elements. But I'm pretty sure that NumPy offers a way to
> handle this more elegantly and much faster.
> 
> As a bonus, I'd be happy if the solution would allow me to compute also
> standard deviation, min, max, etc.
> 
> Thanks in advance for any help!
> Franck
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From aisaac at american.edu  Tue Jan  6 10:11:51 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 06 Jan 2009 10:11:51 -0500
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <49636E5A.2070301@gmail.com>
References: <496325C5.4000505@univ-paris12.fr>	<200901061256.44641.faltet@pytables.org>
	<49636E5A.2070301@gmail.com>
Message-ID: <496374B7.9050900@american.edu>

A Tuesday 06 January 2009, Franck Pommereau escrigu?:
> s = {} # sum of y values for each distinct x (as keys)
> n = {} # number of summed values (same keys)
> for x, y in zip(X, Y) :
>     s[x] = s.get(x, 0.0) + y
>     n[x] = n.get(x, 0) + 1

Maybe this is not so bad with a couple changes?

from collections import defaultdict
from itertools import izip
s = defaultdict(int) # sum of y values for each distinct x (as keys)
n = defaultdict(int) # number of summed values (same keys)
for x, y in izip(x, y) :
     s[x] += y
     n[x] += 1

fwiw,
Alan Isaac




From igorsyl at gmail.com  Tue Jan  6 15:07:56 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Tue, 6 Jan 2009 14:07:56 -0600
Subject: [Numpy-discussion] record array with array elements
Message-ID: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>

Everyone,

Shouldn't the itemsize below be 2?

>>> import numpy as np
>>> dtype = np.dtype( [ (((2,), 'top'), [('nested', 'i1')]) ] )
>>> dtype.itemsize
1
>>> np.__version__
'1.0.4'

The elements of the dtype are of type array of size 2.  Each element is a
(nested) record array of size 2 with one field of type 'i1'.  In contiguous
memory, this should look identical to an 'i1' array of size 2.

-Igor Sylvester
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090106/abfc0a01/attachment.html>

From igorsyl at gmail.com  Tue Jan  6 15:15:07 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Tue, 6 Jan 2009 14:15:07 -0600
Subject: [Numpy-discussion] record array with array elements
In-Reply-To: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>
References: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>
Message-ID: <b3c85b560901061215q555c2382u37e4e2930646edea@mail.gmail.com>

A simpler example returns 1 as well:

np.dtype( [ (((2,), 'a'), 'i1') ] ).itemsize

On Tue, Jan 6, 2009 at 2:07 PM, Igor Sylvester <igorsyl at gmail.com> wrote:

> Everyone,
>
> Shouldn't the itemsize below be 2?
>
> >>> import numpy as np
> >>> dtype = np.dtype( [ (((2,), 'top'), [('nested', 'i1')]) ] )
> >>> dtype.itemsize
> 1
> >>> np.__version__
> '1.0.4'
>
> The elements of the dtype are of type array of size 2.  Each element is a
> (nested) record array of size 2 with one field of type 'i1'.  In contiguous
> memory, this should look identical to an 'i1' array of size 2.
>
> -Igor Sylvester
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090106/b2f096ed/attachment.html>

From robert.kern at gmail.com  Tue Jan  6 16:41:49 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 6 Jan 2009 15:41:49 -0600
Subject: [Numpy-discussion] record array with array elements
In-Reply-To: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>
References: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>
Message-ID: <3d375d730901061341x5af96075h16678408ec33472c@mail.gmail.com>

On Tue, Jan 6, 2009 at 14:07, Igor Sylvester <igorsyl at gmail.com> wrote:
> Everyone,
>
> Shouldn't the itemsize below be 2?
>
>>>> import numpy as np
>>>> dtype = np.dtype( [ (((2,), 'top'), [('nested', 'i1')]) ] )
>>>> dtype.itemsize
> 1
>>>> np.__version__
> '1.0.4'
>
> The elements of the dtype are of type array of size 2.  Each element is a
> (nested) record array of size 2 with one field of type 'i1'.  In contiguous
> memory, this should look identical to an 'i1' array of size 2.

That's not a valid dtype. Array fields should be of the form (name,
subdtype, shape), not ((shape, name), subdtype). I'm not sure why
dtype() does not simply reject this input.

In [22]: np.dtype([('top', [('nested', 'i1')], (2,))]).itemsize
Out[22]: 2

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From reakinator at gmail.com  Tue Jan  6 17:51:11 2009
From: reakinator at gmail.com (Rich E)
Date: Tue, 6 Jan 2009 23:51:11 +0100
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
Message-ID: <a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>

This helped immensely.  I feel like I am getting close to being able
to accomplish what I would like with SWIG: producing a python module
that can be very 'python-like', while co-existing with the c library
that is very 'c-like'.

There is one question still remaining though, is it possible to make
the wrapped function have the same name still?  Using either
my_spectrumMag or spectrumMag means I have to create a number of
inconsistencies between the python module and the c library.  It is
ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
wrapped one, with the same name.  But my attempts at doing this so far
have not compiled because of name conflictions.

Thanks for the help, I think you are doing great things with this
numpy interface/typemaps system.

regards,
Rich


On Tue, Jan 6, 2009 at 3:47 PM, Egor Zindy <ezindy at gmail.com> wrote:
> Hello again,
>
> I really don't know what came over me when I changed your function
> prototype, that wasn't a very thoughtful thing to do!
>
>> Maybe I should look into using an 'insertion block' of code in the
>> interface file, instead of trying to typemap the array?
>
> Insertion blocks... is that %inline code? In which case, yes! Have a look, I
> attached a new version that uses some %inline directives in the
> dftmagnitude.i file.
>
> Basically, you can inline a new function with an easier prototype to wrap.
> The function allocates memory and calls your sms_spectrumMag() function.
>
> my inline function: void my_spectrumMag( int sizeInMag, float *pInRect, int
> *sizeOutMag, float **pOutMag)
>
> there's also a %rename directive: %rename (spectrumMag) my_spectrumMag;
>
> I had a go at defining some exceptions too (no memory and odd number of
> indexes), but I'm not sure errno is the easiest way to go about it...
>
> Hope this helps!
>
> ... and the python test output:
>
> ~>python test_dftmagnitude.py
> array: [1, 1, 2, 2]
> result: [ 1.41421354  2.82842708]
>
> array: [1, 1, 2, 2, 3, 3, 4, 4]
> result: [ 1.41421354  2.82842708  4.2426405   5.65685415]
>
> array: [1, 1, 2]
> result:
> Traceback (most recent call last):
>   File "test_dftmagnitude.py", line 15, in <module>
>     print "result:",dftmagnitude.spectrumMag(a)
> IndexError: Odd number of elements in input array: 3
>
> ~>
>
> Regards,
> Egor
>
>
> On Tue, Jan 6, 2009 at 1:06 AM, Rich E <reakinator at gmail.com> wrote:
>>
>> Egor,
>>
>> Thanks for the help.  I think I want to leave the C code as-is
>> however, as it is perfectly fine there no knowing 'sizeOutMag' because
>> it can deduce both array sizes from one variable.  There are many
>> other similar cases in my code (many where the size of the array is
>> known by a member of a structure passed to the function).
>>
>> Maybe I should look into using an 'insertion block' of code in the
>> interface file, instead of trying to typemap the array?  I am thinking
>> I may just be able to copy the generated code (from SWIG) into my
>> interface file to do this, but I have not tried it yet.
>>
>> I will experiment a little and post again.  Thanks and happy holidays!
>>
>> regards,
>> Rich
>>
>> On Mon, Jan 5, 2009 at 10:42 AM, Egor Zindy <ezindy at gmail.com> wrote:
>> > Hello Rich,
>> >
>> > sorry it took so long to answer back, holidays and all :-)
>> >
>> > That's exactly the kind of SWIG / numpy.i problems I've been working on
>> > over
>> > the past few months: How to generate an array you don't know the size of
>> > a-priori, and then handle the memory deallocation seamlessly. In your
>> > case,
>> > you know that the output array will be half the size of the input array,
>> > but
>> > this falls under the more general case of "not knowing the output size
>> > a-priori".
>> >
>> > Have a look at the files attached. I've rewritten your function header
>> > as:
>> > void sms_spectrumMag( int sizeInMag, float *pInRect, int *sizeOutMag,
>> > float
>> > **pOutMag);
>> >
>> > Easy to see what the input and output arrays are now. Then my numpy.i
>> > handles the memory deallocation of the **pOutMag array.
>> >
>> > I've actually moved my numpy.i explanations to the scipy/numpy cookbook
>> > last
>> > week :-)
>> > http://www.scipy.org/Cookbook/SWIG_Memory_Deallocation
>> >
>> > Hope it all makes sense. If you have any questions, don't hesitate!
>> >
>> >>python test_dftmagnitude.py
>> > [1, 1, 2, 2]
>> > [ 1.41421354  2.82842708]
>> > [1, 1, 2, 2, 3, 3, 4, 4]
>> > [ 1.41421354  2.82842708  4.2426405   5.65685415]
>> > [1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
>> > [ 1.41421354  2.82842708  4.2426405   5.65685415  7.07106781]
>> >
>> > Regards,
>> > Egor
>> >
>> > On Wed, Dec 24, 2008 at 1:52 AM, Rich E <reakinator at gmail.com> wrote:
>> >>
>> >> Hi list,
>> >>
>> >> My question has to do with the Numpy/SWIG typemapping system.
>> >>
>> >> I recently got the typemaps in numpy.i to work on most of my C
>> >> functions that are wrapped using SWIG, if they have arguments of the
>> >> form (int sizeArray, float *pArray).
>> >>
>> >> Now I am trying to figure out how to wrap function that aren't of the
>> >> form, such as the following function:
>> >>
>> >> /*! \brief compute magnitude spectrum of a DFT
>> >>  *
>> >>  * \param sizeMag              size of output Magnitude (half of input
>> >> real FFT)
>> >>  * \param pFReal               pointer to input FFT real array
>> >> (real/imag floats)
>> >>  * \param pFMAg        pointer to float array of magnitude spectrum
>> >>  */
>> >> void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag)
>> >> {
>> >>       int i, it2;
>> >>       float fReal, fImag;
>> >>
>> >>       for (i=0; i<sizeMag; i++)
>> >>       {
>> >>               it2 = i << 1;
>> >>               fReal = pInRect[it2];
>> >>               fImag = pInRect[it2+1];
>> >>               pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag);
>> >>       }
>> >> }
>> >>
>> >> There are two arrays, one is half the size of the other.  But, SWIG
>> >> doesn't know this, according to the type map it will think *pInRect is
>> >> of size sizeMag and will not know anything about *pOutMag.
>> >>
>> >> Ideally in python, I would like to call the function as
>> >> sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of
>> >> nArray2, and nArray2 is of size sizeMag.
>> >>
>> >> I think in order to do this (although if someone has a better
>> >> suggestion, I am open to it), I will have to modify the typemap in
>> >> order to tell SWIG how to call the C function properly.  I do not want
>> >> to have to edit the wrapped C file every time it is regenerated from
>> >> the interface file.
>> >>
>> >>
>> >> Here is a start I made with the existing typemap code in numpy.i (not
>> >> working):
>> >>
>> >> /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>> >>  */
>> >> %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY,
>> >>          fragment="NumPy_Macros")
>> >>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>> >> {
>> >>  $1 = is_array($input) && PyArray_EquivTypenums(array_type($input),
>> >>                                                DATA_TYPECODE);
>> >> }
>> >> %typemap(in,
>> >>        fragment="NumPy_Fragments")
>> >>  (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1)
>> >>  (PyArrayObject* array=NULL, int i=0)
>> >> {
>> >>  array = obj_to_array_no_conversion($input, DATA_TYPECODE);
>> >>  if (!array || !require_dimensions(array,1) ||
>> >> !require_contiguous(array)
>> >>     || !require_native(array)) SWIG_fail;
>> >>  $1 = 1;
>> >>  for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i);
>> >>  $2 = (DATA_TYPE*) array_data(array);
>> >> }
>> >>
>> >> and try to alter it to allow for a conversion of type:
>> >> (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2)
>> >> where ARRAY1 is size DIM1 * 2 and  ARRAY2 is size DIM1.  Then I can
>> >> %apply this to my function that I mentioned in the last post.
>> >>
>> >> So here are my first two questions:
>> >>
>> >> 1) where is DIM1 used to declare the array size?  I don't see where it
>> >> is used at all, and I need to somewhere multiply it by 2 to declare
>> >> the size of ARRAY1
>> >>
>> >> 2) I am not understanding where $input comes from, so I do not
>> >> understand how to distinguish between ARRAY1 and ARRAY2.  In the
>> >> attempt I have already tried, I think I just use the pointer to ARRAY1
>> >> twice.
>> >>
>> >> If anyone has suggestions on how to solve this problem, thanks!
>> >>
>> >> regards,
>> >> Rich
>> >> _______________________________________________
>> >> Numpy-discussion mailing list
>> >> Numpy-discussion at scipy.org
>> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From igorsyl at gmail.com  Tue Jan  6 23:04:49 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Tue, 6 Jan 2009 22:04:49 -0600
Subject: [Numpy-discussion] record array with array elements
In-Reply-To: <3d375d730901061341x5af96075h16678408ec33472c@mail.gmail.com>
References: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>
	<3d375d730901061341x5af96075h16678408ec33472c@mail.gmail.com>
Message-ID: <b3c85b560901062004lac9c510j4b58b02e1d9ea83d@mail.gmail.com>

If array fields should be of the form (name,subdtype, shape), how do I
specify field offsets? My datatype is word-aligned.

Thanks.

On Tue, Jan 6, 2009 at 3:41 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Jan 6, 2009 at 14:07, Igor Sylvester <igorsyl at gmail.com> wrote:
> > Everyone,
> >
> > Shouldn't the itemsize below be 2?
> >
> >>>> import numpy as np
> >>>> dtype = np.dtype( [ (((2,), 'top'), [('nested', 'i1')]) ] )
> >>>> dtype.itemsize
> > 1
> >>>> np.__version__
> > '1.0.4'
> >
> > The elements of the dtype are of type array of size 2.  Each element is a
> > (nested) record array of size 2 with one field of type 'i1'.  In
> contiguous
> > memory, this should look identical to an 'i1' array of size 2.
>
> That's not a valid dtype. Array fields should be of the form (name,
> subdtype, shape), not ((shape, name), subdtype). I'm not sure why
> dtype() does not simply reject this input.
>
> In [22]: np.dtype([('top', [('nested', 'i1')], (2,))]).itemsize
> Out[22]: 2
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090106/f9d4a400/attachment.html>

From robert.kern at gmail.com  Tue Jan  6 23:21:15 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 6 Jan 2009 22:21:15 -0600
Subject: [Numpy-discussion] record array with array elements
In-Reply-To: <b3c85b560901062004lac9c510j4b58b02e1d9ea83d@mail.gmail.com>
References: <b3c85b560901061207y6262a2e6taea3889d373ea5d9@mail.gmail.com>
	<3d375d730901061341x5af96075h16678408ec33472c@mail.gmail.com>
	<b3c85b560901062004lac9c510j4b58b02e1d9ea83d@mail.gmail.com>
Message-ID: <3d375d730901062021v3f3cd03cj3deac5339f51672f@mail.gmail.com>

On Tue, Jan 6, 2009 at 22:04, Igor Sylvester <igorsyl at gmail.com> wrote:
> If array fields should be of the form (name,subdtype, shape), how do I
> specify field offsets? My datatype is word-aligned.

With dtype(some_list), you need to explicitly include the padding.
E.g. ('', '|V4') to add 4 bytes of padding. Alternately, you can use
dtype(some_dict):

dtype(dict(
  names=['x', 'y'],
  formats=['u1', ('u1', (2,2))],
  offsets=[0, 4],
))

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From matthieu.brucher at gmail.com  Wed Jan  7 02:58:56 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 7 Jan 2009 08:58:56 +0100
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
Message-ID: <e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>

2009/1/6 Rich E <reakinator at gmail.com>:
> This helped immensely.  I feel like I am getting close to being able
> to accomplish what I would like with SWIG: producing a python module
> that can be very 'python-like', while co-existing with the c library
> that is very 'c-like'.
>
> There is one question still remaining though, is it possible to make
> the wrapped function have the same name still?  Using either
> my_spectrumMag or spectrumMag means I have to create a number of
> inconsistencies between the python module and the c library.  It is
> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
> wrapped one, with the same name.  But my attempts at doing this so far
> have not compiled because of name conflictions.

Ok course you can. The function is renamed only if you say so. Perhaps
can you provide a small example of what doesn't work at the moment ?

> Thanks for the help, I think you are doing great things with this
> numpy interface/typemaps system.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From pommereau at univ-paris12.fr  Wed Jan  7 07:37:53 2009
From: pommereau at univ-paris12.fr (Franck Pommereau)
Date: Wed, 07 Jan 2009 13:37:53 +0100
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <1231253131.6741.50.camel@sebook>
References: <496325C5.4000505@univ-paris12.fr>
	<1231253131.6741.50.camel@sebook>
Message-ID: <4964A221.9070808@univ-paris12.fr>

Hi all,

First, let me say that I'm impressed: this mailing list is probably the
most reactive I've ever seen. I've asked my first question and got
immediately more solutions than time to test them... Many thanks to all
the answerers.

Using the various proposals, I ran two performance tests:
 - test 1: 2000000 random values
 - test 2: 1328724 values from my real use case

Here are the various functions and how they perform:

def f0 (x, y) :
    """Initial version

    test 1 CPU times: 13.37s
    test 2 CPU times: 5.92s
    """
    s, n = {}, {}
    for a, b in zip(x, y) :
        s[a] = s.get(a, 0.0) + b
        n[a] = n.get(a, 0) + 1
    return (numpy.array([a for a in sorted(s)]),
            numpy.array([s[a]/n[a] for a in sorted(s)]))

def f1 (x, y) :
    """Alan G Isaac <aisaac at american.edu>
    Modified in order to sort the result only once.

    test 1 CPU times: 10.86s
    test 2 CPU times: 2.78s

    defaultdict indeed speeds things up, probably avoiding one of two
    sorts is good also
    """
    s, n = defaultdict(float), defaultdict(int)
    for a, b in izip(x, y) :
        s[a] += b
        n[a] += 1
    new_x = numpy.array([a for a in sorted(s)])
    return (new_x,
            numpy.array([s[a]/n[a] for a in new_x]))

def f2 (x, y) :
    """Francesc Alted <faltet at pytables.org>
    Modified with preallocation of arrays (it appeared faster)

    test 1: killed after more than 10 minutes
    test 2 CPU times: 22.01s

    This result is not surprising as I guess a quadratic complexity: one
    pass for each unique value in x, and presumably one nested pass to
    compute y[x==i]
    """
    u = numpy.unique(x)
    m = numpy.array(range(len(u)))
    for pos, i in enumerate(u) :
        g = y[x == i]
        m[pos] = g.mean()
    return u, m

def f3 (x, y) :
    """Sebastian Stephan Berg <sebastian at sipsolutions.net>
    Modified because I can always work in place.

    test 1 CPU times: 17.43s
    test 2 CPU times: 0.21s

    Adopted! This is definitely the fastest one when using real values.
    I tried to preallocate arrays by setting u=numpy.unique(x) and the
    looping on u, but the result is slower, probably because of unique()

    Compared with f1, its slower on larger arrays of random values. It
    may be explained by a complexity argument: f1 as a linear complexity
    (two passes in sequence) while f3 is probably N log N (a sequence of
    one sort, two passes to set x[:] and y[:] and one loop on each
    distinct value with a nested searchsorted that is probably
    logarithmic). But, real values are far from random, and the sort is
    probably more efficient, as well as the while loop is shorter
    because there are less values.
    """
    s = x.argsort()
    x[:] = x[s]
    y[:] = y[s]
    u, means, start, value = [], [], 0, x[0]
    while True:
        next = x.searchsorted(value, side='right')
        u.append(value)
        means.append(y[start:next].mean())
        if next == len(x):
            break
        value = x[next]
        start = next
    return numpy.array(u), numpy.array(means)

def f4 (x, y) :
    """Jean-Baptiste Rudant <boogaloojb at yahoo.fr>

    test 1 CPU times: 111.21s
    test 2 CPU times: 13.48s

    As Jean-Baptiste noticed, this solution is not very efficient (but
    works almost of-the-shelf).
    """
    recXY = numpy.rec.fromarrays((x, x), names='x, y')
    return matplotlib.mlab.rec_groupby(recXY, ('x',),
                                       (('y', numpy.mean, 'y_avg'),))

A few more remarks.

Sebastian Stephan Berg wrote:
> Just thinking. If the parameters are limited, you may be able to use the
> histogram feature? Doing one histogram with Y as weights, then one
> without weights and calculating the mean from this yourself should be
> pretty speedy I imagine.

I'm afraid I don't know what the histogram function computes. But this
may be something worth to investigate because I think I'll need it later
on in order to smooth my graphs (by plotting mean values on intervals).

Bruce Southey wrote:
> If you use Knuth's one pass approach
>
(http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#III._On-line_algorithm)

> you can write a function to get the min, max, mean and variance/standard
> deviation in a single pass through the array rather than one pass for
> each. I do not know if this will provide any advantage as that will
> probably depend on the size of the arrays.

If I understood well, this algorithm computes the variance of a whole
array, I can see how to adapt it to compute mean (already done by the
algorithm), max, min, etc., but I did not see how it can be adapted to
my case.

> Also, please use the highest precision possible (ie float128) for your
> arrays to minimize numerical error due to the size of your arrays.

Thanks for the advice!

So, thank you again everybody.
Cheers,
Franck


From nicolas.roux at st.com  Wed Jan  7 10:19:16 2009
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Wed, 7 Jan 2009 16:19:16 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
Message-ID: <004401c970db$4e829820$e7ad810a@gnb.st.com>

Hi,

I need help ;-)
I have here a testcase which works much faster in Matlab than Numpy.
 
The following code takes less than 0.9sec in Matlab, but 21sec in Python.
Numpy is 24 times slower than Matlab !
The big trouble I have is a large team of people within my company is ready to replace Matlab by Numpy/Scipy/Matplotlib,
but I have to demonstrate that this kind of Python Code is executed with the same performance than Matlab, without writing C extension.
This is becoming a critical point for us.

This is a testcase that people would like to see working without any code restructuring.
The reasons are:
- this way of writing is fairly natural.
- the original code which showed me the matlab/Numpy performance differences is much more complex,
and can't benefit from broadcasting or other numpy tips (I can later give this code)

...So I really need to use the code below, without restructuring.

Numpy/Python code:
#####################################################################
import numpy
import time

print "Start test \n" 

dim = 3000

a = numpy.zeros((dim,dim,3))

start = time.clock()

for i in range(dim):
    for j in range(dim):
        a[i,j,0] = a[i,j,1]
        a[i,j,2] = a[i,j,0]
        a[i,j,1] = a[i,j,2]

end = time.clock() - start        

print "Test done,   %f sec" % end
#####################################################################

Matlab code:
#####################################################################
'Start test'
dim = 3000;
tic;
a =zeros(dim,dim,3);
for i = 1:dim
    for j = 1:dim 
        a(i,j,1) = a(i,j,2);
        a(i,j,2) = a(i,j,1);
        a(i,j,3) = a(i,j,3);
    end
end
toc
'Test done'
#####################################################################

Any idea on it ?
Did I missed something ?

Thanks a lot, in advance for your help.


Cheers,
Nicolas.



From david at ar.media.kyoto-u.ac.jp  Wed Jan  7 10:16:46 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 08 Jan 2009 00:16:46 +0900
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <004401c970db$4e829820$e7ad810a@gnb.st.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
Message-ID: <4964C75E.6020103@ar.media.kyoto-u.ac.jp>

Nicolas ROUX wrote:
> Hi,
>
> I need help ;-)
> I have here a testcase which works much faster in Matlab than Numpy.
>  
> The following code takes less than 0.9sec in Matlab, but 21sec in Python.
> Numpy is 24 times slower than Matlab !
> The big trouble I have is a large team of people within my company is ready to replace Matlab by Numpy/Scipy/Matplotlib,
> but I have to demonstrate that this kind of Python Code is executed with the same performance than Matlab, without writing C extension.
> This is becoming a critical point for us.
>
> This is a testcase that people would like to see working without any code restructuring.
> The reasons are:
> - this way of writing is fairly natural.
> - the original code which showed me the matlab/Numpy performance differences is much more complex,
> and can't benefit from broadcasting or other numpy tips (I can later give this code)
>
> ...So I really need to use the code below, without restructuring.
>
> Numpy/Python code:
> #####################################################################
> import numpy
> import time
>
> print "Start test \n" 
>
> dim = 3000
>
> a = numpy.zeros((dim,dim,3))
>
> start = time.clock()
>
> for i in range(dim):
>     for j in range(dim):
>         a[i,j,0] = a[i,j,1]
>         a[i,j,2] = a[i,j,0]
>         a[i,j,1] = a[i,j,2]
>
> end = time.clock() - start        
>
> print "Test done,   %f sec" % end
> #####################################################################
>
> Matlab code:
> #####################################################################
> 'Start test'
> dim = 3000;
> tic;
> a =zeros(dim,dim,3);
> for i = 1:dim
>     for j = 1:dim 
>         a(i,j,1) = a(i,j,2);
>         a(i,j,2) = a(i,j,1);
>         a(i,j,3) = a(i,j,3);
>     end
> end
> toc
> 'Test done'
> #####################################################################
>
> Any idea on it ?
> Did I missed something ?
>   

I think on recent versions of matlab, there is nothing you can do
without modifying the code: matlab has some JIT compilation for loops,
which is supposed to speed up those cases - at least, that's what is
claimed by matlab. The above loops are typical examples where this
should work reasonably well I believe:

http://www.mathworks.com/access/helpdesk_r13/help/techdoc/matlab_prog/ch7_pe10.html

If you really have to use loops, then matlab will be faster. But maybe
you don't; can you show us a more typical example ?

cheers,

David


From rmay31 at gmail.com  Wed Jan  7 10:44:50 2009
From: rmay31 at gmail.com (Ryan May)
Date: Wed, 07 Jan 2009 09:44:50 -0600
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <004401c970db$4e829820$e7ad810a@gnb.st.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
Message-ID: <4964CDF2.9090308@gmail.com>

Nicolas ROUX wrote:
> Hi,
> 
> I need help ;-)
> I have here a testcase which works much faster in Matlab than Numpy.
>  
> The following code takes less than 0.9sec in Matlab, but 21sec in Python.
> Numpy is 24 times slower than Matlab !
> The big trouble I have is a large team of people within my company is ready to replace Matlab by Numpy/Scipy/Matplotlib,
> but I have to demonstrate that this kind of Python Code is executed with the same performance than Matlab, without writing C extension.
> This is becoming a critical point for us.
> 
> This is a testcase that people would like to see working without any code restructuring.
> The reasons are:
> - this way of writing is fairly natural.
> - the original code which showed me the matlab/Numpy performance differences is much more complex,
> and can't benefit from broadcasting or other numpy tips (I can later give this code)
> 
> ...So I really need to use the code below, without restructuring.
> 
> Numpy/Python code:
> #####################################################################
> import numpy
> import time
> 
> print "Start test \n" 
> 
> dim = 3000
> 
> a = numpy.zeros((dim,dim,3))
> 
> start = time.clock()
> 
> for i in range(dim):
>     for j in range(dim):
>         a[i,j,0] = a[i,j,1]
>         a[i,j,2] = a[i,j,0]
>         a[i,j,1] = a[i,j,2]
> 
> end = time.clock() - start        
> 
> print "Test done,   %f sec" % end
> #####################################################################
<SNIP>
> Any idea on it ?
> Did I missed something ?

I think you may have reduced the complexity a bit too much.  The python code
above sets all of the elements equal to a[i,j,1].  Is there any reason you can't
use slicing to avoid the loops?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From matthieu.brucher at gmail.com  Wed Jan  7 10:53:36 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 7 Jan 2009 16:53:36 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <004401c970db$4e829820$e7ad810a@gnb.st.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
Message-ID: <e76aa17f0901070753p24203505ld992b299d2fa4b78@mail.gmail.com>

> for i in range(dim):
>    for j in range(dim):
>        a[i,j,0] = a[i,j,1]
>        a[i,j,2] = a[i,j,0]
>        a[i,j,1] = a[i,j,2]

> for i = 1:dim
>    for j = 1:dim
>        a(i,j,1) = a(i,j,2);
>        a(i,j,2) = a(i,j,1);
>        a(i,j,3) = a(i,j,3);
>    end
> end

Hi,

The two loops are not the same.
As David stated, with JIT, the loops may be vectorized by Matlab on the fly.

-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From chaos.proton at gmail.com  Wed Jan  7 10:58:40 2009
From: chaos.proton at gmail.com (Grissiom)
Date: Wed, 7 Jan 2009 23:58:40 +0800
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964CDF2.9090308@gmail.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964CDF2.9090308@gmail.com>
Message-ID: <a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>

On Wed, Jan 7, 2009 at 23:44, Ryan May <rmay31 at gmail.com> wrote:

> Nicolas ROUX wrote:
> > Hi,
> >
> > I need help ;-)
> > I have here a testcase which works much faster in Matlab than Numpy.
> >
> > The following code takes less than 0.9sec in Matlab, but 21sec in Python.
> > Numpy is 24 times slower than Matlab !
> > The big trouble I have is a large team of people within my company is
> ready to replace Matlab by Numpy/Scipy/Matplotlib,
> > but I have to demonstrate that this kind of Python Code is executed with
> the same performance than Matlab, without writing C extension.
> > This is becoming a critical point for us.
> >
> > This is a testcase that people would like to see working without any code
> restructuring.
> > The reasons are:
> > - this way of writing is fairly natural.
> > - the original code which showed me the matlab/Numpy performance
> differences is much more complex,
> > and can't benefit from broadcasting or other numpy tips (I can later give
> this code)
> >
> > ...So I really need to use the code below, without restructuring.
> >
> > Numpy/Python code:
> > #####################################################################
> > import numpy
> > import time
> >
> > print "Start test \n"
> >
> > dim = 3000
> >
> > a = numpy.zeros((dim,dim,3))
> >
> > start = time.clock()
> >
> > for i in range(dim):
> >     for j in range(dim):
> >         a[i,j,0] = a[i,j,1]
> >         a[i,j,2] = a[i,j,0]
> >         a[i,j,1] = a[i,j,2]
> >
> > end = time.clock() - start
> >
> > print "Test done,   %f sec" % end
> > #####################################################################
> <SNIP>
> > Any idea on it ?
> > Did I missed something ?
>
> I think you may have reduced the complexity a bit too much.  The python
> code
> above sets all of the elements equal to a[i,j,1].  Is there any reason you
> can't
> use slicing to avoid the loops?
>
>
Yes, I think so. I think the testcase  is a matter of python loop vs matlab
loop rather than python vs matlab.

-- 
Cheers,
Grissiom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090107/f9494024/attachment.html>

From jdh2358 at gmail.com  Wed Jan  7 11:14:44 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Wed, 7 Jan 2009 10:14:44 -0600
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <4964A221.9070808@univ-paris12.fr>
References: <496325C5.4000505@univ-paris12.fr>
	<1231253131.6741.50.camel@sebook> <4964A221.9070808@univ-paris12.fr>
Message-ID: <88e473830901070814p7d4fe368t2b908e01ab44b9ea@mail.gmail.com>

On Wed, Jan 7, 2009 at 6:37 AM, Franck Pommereau
<pommereau at univ-paris12.fr> wrote:

> def f4 (x, y) :
>    """Jean-Baptiste Rudant <boogaloojb at yahoo.fr>
>
>    test 1 CPU times: 111.21s
>    test 2 CPU times: 13.48s
>
>    As Jean-Baptiste noticed, this solution is not very efficient (but
>    works almost of-the-shelf).
>    """
>    recXY = numpy.rec.fromarrays((x, x), names='x, y')
>    return matplotlib.mlab.rec_groupby(recXY, ('x',),
>                                       (('y', numpy.mean, 'y_avg'),))

This probably will have no impact on your tests, but this looks like a
bug.  You probably mean:

  recXY = numpy.rec.fromarrays((x, y), names='x, y')

Could you post the code you use to generate you inputs (ie what is x?)

I will look into trying some of the suggestions here to improve the
performance on rec_groupby.  One thing that slows it down is that it
supports an arbitrary number of keys -- eg groupby ('year', 'month')
-- whereas the examples above are using a single value lookup.

JDH


From pommereau at univ-paris12.fr  Wed Jan  7 11:33:54 2009
From: pommereau at univ-paris12.fr (Franck Pommereau)
Date: Wed, 07 Jan 2009 17:33:54 +0100
Subject: [Numpy-discussion] [Newbie] Fast plotting
In-Reply-To: <88e473830901070814p7d4fe368t2b908e01ab44b9ea@mail.gmail.com>
References: <496325C5.4000505@univ-paris12.fr>	<1231253131.6741.50.camel@sebook>
	<4964A221.9070808@univ-paris12.fr>
	<88e473830901070814p7d4fe368t2b908e01ab44b9ea@mail.gmail.com>
Message-ID: <4964D972.80502@univ-paris12.fr>

> This probably will have no impact on your tests, but this looks like a
> bug.  You probably mean:
> 
>   recXY = numpy.rec.fromarrays((x, y), names='x, y')

Sure! Thanks.

> Could you post the code you use to generate you inputs (ie what is x?)

My code is probably not usable by somebody else than me. I'm presently
too busy to clean it and add comments. But as soon as I'll be able to do
so, I'll send you the usable version.

Cheers,
Franck




From josef.pktd at gmail.com  Wed Jan  7 11:36:27 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 7 Jan 2009 11:36:27 -0500
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964CDF2.9090308@gmail.com>
	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>
Message-ID: <1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>

On Wed, Jan 7, 2009 at 10:58 AM, Grissiom <chaos.proton at gmail.com> wrote:
> On Wed, Jan 7, 2009 at 23:44, Ryan May <rmay31 at gmail.com> wrote:
>>
>> Nicolas ROUX wrote:
>> > Hi,
>> >
>> > I need help ;-)
>> > I have here a testcase which works much faster in Matlab than Numpy.
>> >
>> > The following code takes less than 0.9sec in Matlab, but 21sec in
>> > Python.
>> > Numpy is 24 times slower than Matlab !
>> > The big trouble I have is a large team of people within my company is
>> > ready to replace Matlab by Numpy/Scipy/Matplotlib,
>> > but I have to demonstrate that this kind of Python Code is executed with
>> > the same performance than Matlab, without writing C extension.
>> > This is becoming a critical point for us.
>> >
>> > This is a testcase that people would like to see working without any
>> > code restructuring.
>> > The reasons are:
>> > - this way of writing is fairly natural.
>> > - the original code which showed me the matlab/Numpy performance
>> > differences is much more complex,
>> > and can't benefit from broadcasting or other numpy tips (I can later
>> > give this code)
>> >
>> > ...So I really need to use the code below, without restructuring.
>> >
>> > Numpy/Python code:
>> > #####################################################################
>> > import numpy
>> > import time
>> >
>> > print "Start test \n"
>> >
>> > dim = 3000
>> >
>> > a = numpy.zeros((dim,dim,3))
>> >
>> > start = time.clock()
>> >
>> > for i in range(dim):
>> >     for j in range(dim):
>> >         a[i,j,0] = a[i,j,1]
>> >         a[i,j,2] = a[i,j,0]
>> >         a[i,j,1] = a[i,j,2]
>> >
>> > end = time.clock() - start
>> >
>> > print "Test done,   %f sec" % end
>> > #####################################################################
>> <SNIP>
>> > Any idea on it ?
>> > Did I missed something ?
>>
>> I think you may have reduced the complexity a bit too much.  The python
>> code
>> above sets all of the elements equal to a[i,j,1].  Is there any reason you
>> can't
>> use slicing to avoid the loops?
>>
>
> Yes, I think so. I think the testcase  is a matter of python loop vs matlab
> loop rather than python vs matlab.
>
> --
> Cheers,
> Grissiom
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>

I tried with matlab 2006a, I don't know if there is JIT, but the main
speed difference comes with the numpy array access.

The test is actually biased in favor of python, since in the matlab
code the initialization with zeros is inside the time count, but
outside in the python version

If I just put b=1.0 inside the double loop (no numpy)

Python  1.453644 sec
matlab  0.335249 seconds, with zeros outside loop:  0.060582 seconds

with original array assignment:

python/numpy 32.745030 sec
matlab             1.633415 seconds, with zeros outside loop: 1.251597 seconds

(putting the loop in a function and using psyco reduces speed by 30%)


Josef


From reakinator at gmail.com  Wed Jan  7 11:43:07 2009
From: reakinator at gmail.com (Rich E)
Date: Wed, 7 Jan 2009 17:43:07 +0100
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
	<e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
Message-ID: <a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>

Here is my example, trying to wrap the function sms_spectrumMag that
we have been dealing with:

%apply (int DIM1, float* IN_ARRAY1) {(int sizeInArray, float* pInArray)};
%apply (int DIM1, float* INPLACE_ARRAY1) {(int sizeOutArray, float* pOutArray)};

%inline %{

        void my_spectrumMag( int sizeInArray, float *pInArray, int
sizeOutArray, float *pOutArray)
        {
                sms_spectrumMag(sizeOutArray, pInArray, pOutArray);
        }

%}


at this point,  have the new function my_spectrumMag that wraps
sms_spectrumMag() and provides arguments that can be typemapped using
numpy.i  Now, I don't want to have to call the function
my_spectrumMag() in python, I want to use the original name, I would
like to call the function as:

sms_spectrumMag(numpyArray1, numpyArray2)

But, trying to %rename my_spectrumMag to sms_spectrumMag does not
work, the original sms_spectrumMag gets called in python instead.
Trying to %ignore the original function first as follows removes the
sms_spectrumMag completely from the module and I am left with
my_spectrumMag:

%ignore sms_spectrumMag;
%rename (sms_spectrumMag) my_spectrumMag;


Do you see my problem?


On Wed, Jan 7, 2009 at 8:58 AM, Matthieu Brucher
<matthieu.brucher at gmail.com> wrote:
> 2009/1/6 Rich E <reakinator at gmail.com>:
>> This helped immensely.  I feel like I am getting close to being able
>> to accomplish what I would like with SWIG: producing a python module
>> that can be very 'python-like', while co-existing with the c library
>> that is very 'c-like'.
>>
>> There is one question still remaining though, is it possible to make
>> the wrapped function have the same name still?  Using either
>> my_spectrumMag or spectrumMag means I have to create a number of
>> inconsistencies between the python module and the c library.  It is
>> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
>> wrapped one, with the same name.  But my attempts at doing this so far
>> have not compiled because of name conflictions.
>
> Ok course you can. The function is renamed only if you say so. Perhaps
> can you provide a small example of what doesn't work at the moment ?
>
>> Thanks for the help, I think you are doing great things with this
>> numpy interface/typemaps system.
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From josef.pktd at gmail.com  Wed Jan  7 12:27:07 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 7 Jan 2009 12:27:07 -0500
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964CDF2.9090308@gmail.com>
	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>
	<1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>
Message-ID: <1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>

A test case closer to my applications is calling functions in loops:

Python
-----------------------------------
def assgn(a,i,j):
    a[i,j,0] = a[i,j,1] + 1.0
    a[i,j,2] = a[i,j,0]
    a[i,j,1] = a[i,j,2]
    return a

print "Start test \n"

dim = 300#0

a = numpy.zeros((dim,dim,3))
start = time.clock()

for i in range(dim):
    for j in range(dim):
        assgn(a,i,j)

end = time.clock() - start
assert numpy.max(a)==1.0  #added to check inplace substitution
print "Test done,   %f sec" % end
---------------------------

matlab:
------------------------------------------------------------------
function a = tryloopspeed()

'Start test'
dim = 300;
a = zeros(dim,dim,3);
tic;
for i = 1:dim
   for j = 1:dim
        a = assgn(a,i,j);
   end
end
toc
'Test done'
end
function a = assgn(a,i,j)
         a(i,j,1) = a(i,j,2);
         a(i,j,2) = a(i,j,1);
         a(i,j,3) = a(i,j,3);
end
---------------------------

Note: I had to reduce the size of the matrix because I got impatient
waiting for matlab
time:

python: Test done,   0.486127 sec
matlab:
>> output = tryloopspeed();
ans =
Start test
Elapsed time is 511.815971 seconds.
ans =
Test done
>> 511.815971/60.0  #minutes
ans =
          8.530

matlab takes 1053 times the time of python
The problem is that at least in my version of matlab, it copies
function arguments when they are modified. It's possible to work
around this, but not very clean.

So for simple loops python looses, but for other things, python wins
by a huge margin. Unless somebody can spot a mistake in my timing.

Josef


From Chris.Barker at noaa.gov  Wed Jan  7 12:51:58 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 07 Jan 2009 09:51:58 -0800
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964C75E.6020103@ar.media.kyoto-u.ac.jp>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964C75E.6020103@ar.media.kyoto-u.ac.jp>
Message-ID: <4964EBBE.7020706@noaa.gov>

> Nicolas ROUX wrote:
>> The big trouble I have is a large team of people within my company is ready to replace Matlab by Numpy/Scipy/Matplotlib,

we like that!

>> This is a testcase that people would like to see working without any code restructuring.
>> The reasons are:
>> - this way of writing is fairly natural.

Only if you haven't wrapped your brain around array-oriented 
programming! (see below)

>> - the original code which showed me the matlab/Numpy performance differences is much more complex,
>> and can't benefit from broadcasting or other numpy tips (I can later give this code)

so you're asking: "how can I make this code faster without changing it?" 
The only way to do that is to change python or numpy, and while it might 
be nice to do that to improve performance in this type of case, it's a 
tall order!

It's really not a good goal, anyway -- python/numpy is by no means a 
drop-in replacement for MATLAB -- they are very different beasts. 
Personally, I think most of the differences favor Python, but if you try 
to write python the same way you'd write MATLAB, you'll lose most of the 
benefits -- you might as well stick with MATLAB.

However, in this case, MATLAB was traditionally slow with loops and 
indexing and needed to be vectorized for decent performance as well.

It look like they now have a nice JIT compiler for this sort of thing -- 
to get a similar effect in numpy, you'll need to use weave or Cython or 
something, notable not as easy as having the interpreter just do it for you.

I'd love to see a numpy-aware psyco some day, an maybe the new buffer 
interface will facilitate that, but it's inherently harder with numpy -- 
MATLAB at least used to be limited to 2-d arrays of doubles, so far less 
special casing to be done.

Even with this nifty JIT, I think Python has many advantages -- if your 
code is well written, there will be a only a few places with these sorts 
of performance bottlenecks, and weave or Cython, or SWIG, or Ctypes, or 
f2py can all give you a good solution.

One other thought -- could numexp help here?


About array-oriented programming:

All lot of folks seem to think that the only reason to "vectorize" code 
in MATLAB, numpy, etc, is for better performance. If MATLAB now has a 
good JIT, then there is no point -- I think that's a mistake. If you 
write your code to work with arrays of data, you get more compact, less 
bug-prone code than if you are working with indexed elements all the 
time. I also think the code is clearer most of the time. I say most, 
because sometimes you do need to do "tricks" to vectorize that can 
obfuscate the code.

I understand that this may be a simplified example, and the real 
use-case could be quite different. However:

>> a = numpy.zeros((dim,dim,3))

so we essentially have three square arrays stacked together -- what do 
they represent? that might help guide you, but without that, I can still 
see:

>> for i in range(dim):
>>     for j in range(dim):

this really means -- for every element of the 2-d arrays, which can be 
written as: a[:,:]

>>         a[i,j,0] = a[i,j,1]
>>         a[i,j,2] = a[i,j,0]
>>         a[i,j,1] = a[i,j,2]

and this is simply swapping the three around. So, if you start out 
thinking in terms of a set of 2-d arrays, rather than a huge pile of 
elements, the code you will arrive at is more like:

a[:,:,0] = a[:,:,1]
a[:,:,2] = a[:,:,0]
a[:,:,1] = a[:,:,2]

With no loops:

or you could give them names:

a0 = a[:,:,0]
a1 = a[:,:,1]
a2 = a[:,:,2]

then:

a0[:] = a1
a2[:] = a0
a1[:] = a2

which, of course, is really:

a[:,:,:] = a1.reshape((dim,dim,1))

but I suspect that that's the result of a typo.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Wed Jan  7 12:56:26 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 07 Jan 2009 09:56:26 -0800
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964CDF2.9090308@gmail.com>
	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>
	<1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>
	<1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>
Message-ID: <4964ECCA.7060103@noaa.gov>

josef.pktd at gmail.com wrote:
> So for simple loops python looses, but for other things, python wins
> by a huge margin.

which emphasizes the point that you can't write code the same way in the 
two languages, though I'd argue that that code needs refactoring in any 
language!

However, numpy's reference semantics is definitely a strong advantage of 
MATLAB -- more flexibility in general.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From sturla at molden.no  Wed Jan  7 13:16:51 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 07 Jan 2009 19:16:51 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964C75E.6020103@ar.media.kyoto-u.ac.jp>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964C75E.6020103@ar.media.kyoto-u.ac.jp>
Message-ID: <4964F193.9060801@molden.no>

On 1/7/2009 4:16 PM, David Cournapeau wrote:

> I think on recent versions of matlab, there is nothing you can do
> without modifying the code: matlab has some JIT compilation for loops,
> which is supposed to speed up those cases - at least, that's what is
> claimed by matlab. 

Yes it does. After using both for more than 10 years, my impression is this:

- Matlab slicing creates new arrays. NumPy slicing creates views. NumPy 
is faster and more memory efficient.

- Matlab JIT compiles loops. NumPy does not. Matlab is faster for stupid 
programmers that don't know how use slices. But neither Matlab nor 
Python/NumPy is meant to be used like Java.

- Python has psyco. It is about as good as Matlab's JIT. But psyco has 
no knowledge of NumPy ndarrays.

- Using Cython is easier than writing Matlab MEX files.

- Python has better support for data structures, better built-in 
structures (tuple, lists, dics, sets), and general purpose libraries. 
Matlab has extensive numerical toolboxes that you can buy.

- Matlab pass function arguments by value (albeit COW optimized). Python 
pass references. This makes NumPy more efficient if you need to pass 
large arrays or array slices.

- Matlab tends to fragment the heap (hence the pack command). 
Python/NumPy does not. This makes long-running processes notoriously 
unstable on Matlab.

- Matlab has some numerical libraries that are better.

- I like the Matlab command prompt and IDE better. But its not enough to 
make me want to use it.

- Python is a proper programming language. Matlab is a numerical 
scripting language - good for small scripts but not complex software 
systems.


Sturla Molden









From xavier.gnata at gmail.com  Wed Jan  7 13:29:41 2009
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Wed, 07 Jan 2009 19:29:41 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964F193.9060801@molden.no>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>	<4964C75E.6020103@ar.media.kyoto-u.ac.jp>
	<4964F193.9060801@molden.no>
Message-ID: <4964F495.5070200@gmail.com>

Well it is the best pitch for numpy versus matlab I have read so far :)
(and I 100% agree)

Xavier
> On 1/7/2009 4:16 PM, David Cournapeau wrote:
>
>   
>> I think on recent versions of matlab, there is nothing you can do
>> without modifying the code: matlab has some JIT compilation for loops,
>> which is supposed to speed up those cases - at least, that's what is
>> claimed by matlab. 
>>     
>
> Yes it does. After using both for more than 10 years, my impression is this:
>
> - Matlab slicing creates new arrays. NumPy slicing creates views. NumPy 
> is faster and more memory efficient.
>
> - Matlab JIT compiles loops. NumPy does not. Matlab is faster for stupid 
> programmers that don't know how use slices. But neither Matlab nor 
> Python/NumPy is meant to be used like Java.
>
> - Python has psyco. It is about as good as Matlab's JIT. But psyco has 
> no knowledge of NumPy ndarrays.
>
> - Using Cython is easier than writing Matlab MEX files.
>
> - Python has better support for data structures, better built-in 
> structures (tuple, lists, dics, sets), and general purpose libraries. 
> Matlab has extensive numerical toolboxes that you can buy.
>
> - Matlab pass function arguments by value (albeit COW optimized). Python 
> pass references. This makes NumPy more efficient if you need to pass 
> large arrays or array slices.
>
> - Matlab tends to fragment the heap (hence the pack command). 
> Python/NumPy does not. This makes long-running processes notoriously 
> unstable on Matlab.
>
> - Matlab has some numerical libraries that are better.
>
> - I like the Matlab command prompt and IDE better. But its not enough to 
> make me want to use it.
>
> - Python is a proper programming language. Matlab is a numerical 
> scripting language - good for small scripts but not complex software 
> systems.
>
>
> Sturla Molden
>
>
>
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From faltet at pytables.org  Wed Jan  7 13:31:06 2009
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 7 Jan 2009 19:31:06 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964EBBE.7020706@noaa.gov>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964C75E.6020103@ar.media.kyoto-u.ac.jp>
	<4964EBBE.7020706@noaa.gov>
Message-ID: <200901071931.06474.faltet@pytables.org>

A Wednesday 07 January 2009, Christopher Barker escrigu?:
[clip]
> Even with this nifty JIT, I think Python has many advantages -- if
> your code is well written, there will be a only a few places with
> these sorts of performance bottlenecks, and weave or Cython, or SWIG,
> or Ctypes, or f2py can all give you a good solution.

Agreed.  Specially Cython, with the latests improvements for supporting 
optimized NumPy indexing:

http://wiki.cython.org/tutorials/numpy

would make these loops to work much faster.

> One other thought -- could numexp help here?

I don't think so.  Numexpr is for computing expresions like 'a-b**3-c' 
element-wise (a, b and c are arrays) quickly.  The main reason of its 
high performance is that it avoids temporary copies of intermediate 
results.  In order to use it, you need to vectorize first your loops, 
and this is not what Nicolas wants.

Cheers,

-- 
Francesc Alted


From sturla at molden.no  Wed Jan  7 13:32:58 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 07 Jan 2009 19:32:58 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964ECCA.7060103@noaa.gov>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>	<4964CDF2.9090308@gmail.com>	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>	<1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>	<1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>
	<4964ECCA.7060103@noaa.gov>
Message-ID: <4964F55A.4070108@molden.no>

On 1/7/2009 6:56 PM, Christopher Barker wrote:

>> So for simple loops python looses, but for other things, python wins
>> by a huge margin.
> 
> which emphasizes the point that you can't write code the same way in the 
> two languages, though I'd argue that that code needs refactoring in any 
> language!

Roux example would be bad in either language. Slices ('vectorization' in 
Matlab lingo) is preferred in both cases. It's just that neither Matlab 
nor Python/NumPy was designed to be used like Java. For loops should not 
be abused in Python nor in Matlab (but Matlab is more forgiving now than 
it used to be).


Sturla Molden


From sturla at molden.no  Wed Jan  7 13:50:54 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 07 Jan 2009 19:50:54 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964EBBE.7020706@noaa.gov>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>	<4964C75E.6020103@ar.media.kyoto-u.ac.jp>
	<4964EBBE.7020706@noaa.gov>
Message-ID: <4964F98E.5060906@molden.no>

On 1/7/2009 6:51 PM, Christopher Barker wrote:

> Even with this nifty JIT, 

It is not a very nifty JIT. It can transform some simple loops into 
vectorized expressions. And it removes the overhead from indexing with 
doubles.

But if you are among those that do

n = length(x)
m = 0
for i = 1.0 : n
   m = m + x(i)
end
m = m / n

instead of

m = mean(x)

it will be nifty enough.


> All lot of folks seem to think that the only reason to "vectorize" code 
> in MATLAB, numpy, etc, is for better performance. If MATLAB now has a 
> good JIT, then there is no point -- I think that's a mistake. 

Fortran 90/95 has array slicing as well.


Sturla Molden








From josef.pktd at gmail.com  Wed Jan  7 13:52:43 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 7 Jan 2009 13:52:43 -0500
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964F55A.4070108@molden.no>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964CDF2.9090308@gmail.com>
	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>
	<1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>
	<1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>
	<4964ECCA.7060103@noaa.gov> <4964F55A.4070108@molden.no>
Message-ID: <1cd32cbb0901071052x1f3c129cl5c2239987c047fc1@mail.gmail.com>

On Wed, Jan 7, 2009 at 1:32 PM, Sturla Molden <sturla at molden.no> wrote:
> On 1/7/2009 6:56 PM, Christopher Barker wrote:
>
>>> So for simple loops python looses, but for other things, python wins
>>> by a huge margin.
>>
>> which emphasizes the point that you can't write code the same way in the
>> two languages, though I'd argue that that code needs refactoring in any
>> language!
>
> Roux example would be bad in either language. Slices ('vectorization' in
> Matlab lingo) is preferred in both cases. It's just that neither Matlab
> nor Python/NumPy was designed to be used like Java. For loops should not
> be abused in Python nor in Matlab (but Matlab is more forgiving now than
> it used to be).
>
>
> Sturla Molden
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

I'm missing name spaces in matlab. everything is
from path import *

and it's more difficult to keep are larger project organized in matlab
than in python.

But, I think,
matlab is ahead in parallelization (which I haven't used much)
and learning matlab is easier than numpy. (dtypes and broadcasting are
more restrictive in matlab but, for a beginner, easier to figure out)

Josef


From sturla at molden.no  Wed Jan  7 14:26:25 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 07 Jan 2009 20:26:25 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <1cd32cbb0901071052x1f3c129cl5c2239987c047fc1@mail.gmail.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>	<4964CDF2.9090308@gmail.com>	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>	<1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>	<1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>	<4964ECCA.7060103@noaa.gov>
	<4964F55A.4070108@molden.no>
	<1cd32cbb0901071052x1f3c129cl5c2239987c047fc1@mail.gmail.com>
Message-ID: <496501E1.2060400@molden.no>

On 1/7/2009 7:52 PM, josef.pktd at gmail.com wrote:

> But, I think,
> matlab is ahead in parallelization (which I haven't used much)

Not really. There is e.g. nothing like Python's multiprocessing package 
in Matlab. Matlab is genrally single-threaded. Python is multi-threaded 
but there is a GIL. And having multiple Matlab processes running 
simultaneously consumes a lot of resources. Python is far better in this 
respect. Don't confuse vectorization with parallelization. It is not the 
same. If you are going to do real parallelization, you are better off 
using Python with multiprocessing or mpi4py.


> and learning matlab is easier than numpy. (dtypes and broadcasting are
> more restrictive in matlab but, for a beginner, easier to figure out)

The available data types is about the same, at least last time I 
checked. (I am not thinking about Python built-ins here, but NumPy dtypes.)

Matlab does not have broadcasting. Array shapes must always match.


S.M.



From robert.kern at gmail.com  Wed Jan  7 15:54:50 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 7 Jan 2009 15:54:50 -0500
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <004401c970db$4e829820$e7ad810a@gnb.st.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
Message-ID: <3d375d730901071254p3a3a110nccf165434a37f5a0@mail.gmail.com>

On Wed, Jan 7, 2009 at 10:19, Nicolas ROUX <nicolas.roux at st.com> wrote:
> Hi,
>
> I need help ;-)
> I have here a testcase which works much faster in Matlab than Numpy.
>
> The following code takes less than 0.9sec in Matlab, but 21sec in Python.
> Numpy is 24 times slower than Matlab !
> The big trouble I have is a large team of people within my company is ready to replace Matlab by Numpy/Scipy/Matplotlib,
> but I have to demonstrate that this kind of Python Code is executed with the same performance than Matlab, without writing C extension.
> This is becoming a critical point for us.
>
> This is a testcase that people would like to see working without any code restructuring.

Basically, if you want efficient numpy code, you have to use numpy
idioms. If you want to continue to use Matlab idioms, keep using
Matlab.

> The reasons are:
> - this way of writing is fairly natural.
> - the original code which showed me the matlab/Numpy performance differences is much more complex,
> and can't benefit from broadcasting or other numpy tips (I can later give this code)

Please do. Otherwise, we can't actually address your concerns.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dwf at cs.toronto.edu  Wed Jan  7 16:41:53 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 7 Jan 2009 16:41:53 -0500
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <496501E1.2060400@molden.no>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>	<4964CDF2.9090308@gmail.com>	<a0adeea50901070758i4c44a2d3ub6b39559fd8f5ac7@mail.gmail.com>	<1cd32cbb0901070836n56aa3550h8bedf8aad186a581@mail.gmail.com>	<1cd32cbb0901070927v43e56df2x44e5e764ea9a29a7@mail.gmail.com>	<4964ECCA.7060103@noaa.gov>
	<4964F55A.4070108@molden.no>
	<1cd32cbb0901071052x1f3c129cl5c2239987c047fc1@mail.gmail.com>
	<496501E1.2060400@molden.no>
Message-ID: <877CE7F7-8268-4BDD-8503-62F7789608A9@cs.toronto.edu>

On 7-Jan-09, at 2:26 PM, Sturla Molden wrote:

> Matlab does not have broadcasting. Array shapes must always match.

Not totally true. They introduced a clunky, clunky syntax for it in  
version 7, IIRC, called 'bsxfun'.

See http://tinyurl.com/9e7kyt . It's a better solution than indexing  
with a huge ones() or repmat()'ing all over the place, but not nearly  
as nice as NumPy overloading.

David


From gael.varoquaux at normalesup.org  Wed Jan  7 17:50:38 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 7 Jan 2009 23:50:38 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4964F495.5070200@gmail.com>
References: <004401c970db$4e829820$e7ad810a@gnb.st.com>
	<4964C75E.6020103@ar.media.kyoto-u.ac.jp>
	<4964F193.9060801@molden.no> <4964F495.5070200@gmail.com>
Message-ID: <20090107225038.GB5186@phare.normalesup.org>

On Wed, Jan 07, 2009 at 07:29:41PM +0100, Xavier Gnata wrote:
> Well it is the best pitch for numpy versus matlab I have read so far :)
> (and I 100% agree)

+1. This is an excellent text. IMHO it should be on the wiki somewhere.

Ga?l


From jdh2358 at gmail.com  Wed Jan  7 17:52:50 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Wed, 7 Jan 2009 16:52:50 -0600
Subject: [Numpy-discussion] my cython is slow
Message-ID: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>

Partly as an excuse to learn cython, and partly because I need to eke
out some extra performance of a neighborhood search, I tried to code
up a brute force neighborhood search in cython around an N-dimensional
point p.  I need to incrementally add a point, do a search, add
another point, do another search, so some of the algorithms like those
in scipy.stats.spatial which assume a static data structure with lots
of searches over it probably won't help me.

I wrote some cython code to grow a Npoints x Ndimensions numpy array
(doubling in size every time I exceed Npoints) and then doing a brute
force request for all points within a radius r using a euclidean
distance.  The code is working fine, but it is still slower than a
simple numpy implementation (damn you numpy performance!)

Here is the numpy code::

    def find_neighbors_numpy(data, point, radius):
        """
        do a plain ol numpy lookup to compare performance and output

          *data* is a numpoints x numdims array
          *point* is a numdims length vector
          radius is the max distance distance

        return an array of indices into data which are within radius
        """
        numpoints, n = data.shape
        distance = data - point
        r = np.sqrt((distance*distance).sum(axis=1))
        return np.nonzero(r<=radius)[0]

This requires 6 passed through the data, so I should be able to beat
it with a properly crafted cython algorithm.

I have a class NNBF (Nearest Neighbor Brute Force) with an interface like

    NUMDIM = 6
    nn = nnbf.NNBF(NUMDIM)

    print 'loading data... this could take a while'
    # this could take a while
    for i in range(200000):
        x = np.random.rand(NUMDIM)
        nn.add(x)

    x = np.random.rand(NUMDIM)
    radius = 0.2
    ind = nn.find_neighbors(x, radius)

(in my real use case I would be doing a search after every add)

when I run this vs numpy, numpy is a little faster

    testing nnbf...
        10 trials: mean=0.0420, min=0.0400
    testing numpy...
        10 trials: mean=0.0420, min=0.0300


You can grab the code, the python prototype, the test case and setup
file at http://matplotlib.sf.net/nnbf.zip.  You will need a fairly
recent cython to build it:
http://www.cython.org/Cython-0.10.3.tar.gz::

  # build the extension and run the test code
  wget http://matplotlib.sf.net/nnbf.zip
  unzip nnbf.zip
  cd nnbf
  python setup.py build_ext --inplace
  python test_nnbf.py


I'm pasting the cython code nnbf.pyx below.  If anyone can advise me
as to how to remove the bottlenecks (I've decorated the code with some
XXX questions) that would be great.  I've read the tutorial at
http://wiki.cython.org/tutorials/numpy but I think I am still missing
something important.  I tried to follow Anne's code in ckdtree.pyx
which makes use of a raw_data structure but I am not sure how/if to
incorporate this into my code -- perhaps there is some juice there....

"""
A brute force nearest neighbor routine with incremental add.  The
internal array data structure grows as you add points
"""

import numpy as np
cimport numpy as np

cdef extern from "math.h":
     float sqrt(float)

cdef inline int is_neighbor(int n, double*row, double*pp, double d2max):
    """
    return 1 if the sum-of-squares of n length array row[j]-pp[j] <= d2max
    """
    cdef int j
    cdef double d, d2

    d2 = 0.

    for j in range(n):
        d = row[j] - pp[j]
        d2 += d*d
        if d2>d2max:
            return 0
    return 1

cdef class NNBF:
    cdef readonly object data
    #cdef double* raw_data
    cdef readonly int n, numrows, numpoints

    def __init__(self, n):
        """
        create a buffer to hold n dimensional points
        """
        #cdef np.ndarray[double, ndim=2] inner_data


        self.n = n
        self.numrows = 100
        #  XXX how to create empty as contiguous w/o copy?
        self.data = np.empty((self.numrows, self.n), dtype=np.float)
        #inner_data = self.data
        #self.raw_data = <double*>inner_data.data
        self.numpoints = 0


    def add(NNBF self, object point):
        """
        add a point to the buffer, grow if necessary
        """
        #cdef np.ndarray[double, ndim=2] inner_data
        cdef np.ndarray[double, ndim=1] pp
        pp = np.asarray(point).astype(np.float)


        self.data[self.numpoints] = pp
        self.numpoints += 1
        if self.numpoints==self.numrows:
            ## XXX do I need to do memory management here, eg free
            ## raw_data if I were using it?
            self.numrows *= 2
            newdata = np.empty((self.numrows, self.n), np.float)
            newdata[:self.numpoints] = self.data
            self.data = newdata
            #self.raw_data = <double*>inner_data.data

    def get_data(NNBF self):
        """
        return a copy of data added so far as a numpoints x n array
        """
        return self.data[:self.numpoints]


    def find_neighbors(NNBF self, object point, double radius):
        """
        return a list of indices into data which are within radius
        from point
        """
        cdef int i, neighbor
        cdef double d2max
        cdef np.ndarray[double, ndim=1] pp
        cdef np.ndarray[double, ndim=1] row

        if len(point)!=self.n:
            raise ValueError('Expected a length %d vector'%self.n)

        pp = np.asarray(point).astype(np.float)

        d2max = radius*radius
        neighbors = []


        for i in range(self.numpoints):
            # XXX : is there a more efficient way to access the row
            # data?  Can/should we be using raw_data here?
            row = self.data[i]
            neighbor = is_neighbor(
                self.n,
                <double*>row.data,
                <double*>pp.data,
                d2max)

            # if the number of points in the cluster is small, the
            # python list performance should not kill us
            if neighbor:
                neighbors.append(i)

        return neighbors


From efiring at hawaii.edu  Wed Jan  7 18:37:56 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 07 Jan 2009 13:37:56 -1000
Subject: [Numpy-discussion] my cython is slow
In-Reply-To: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>
References: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>
Message-ID: <49653CD4.90808@hawaii.edu>

John Hunter wrote:
> Partly as an excuse to learn cython, and partly because I need to eke
> out some extra performance of a neighborhood search, I tried to code
> up a brute force neighborhood search in cython around an N-dimensional
> point p.  I need to incrementally add a point, do a search, add
> another point, do another search, so some of the algorithms like those
> in scipy.stats.spatial which assume a static data structure with lots
> of searches over it probably won't help me.
> 
> I wrote some cython code to grow a Npoints x Ndimensions numpy array
> (doubling in size every time I exceed Npoints) and then doing a brute
> force request for all points within a radius r using a euclidean
> distance.  The code is working fine, but it is still slower than a
> simple numpy implementation (damn you numpy performance!)

A couple small changes speed it up quite a bit:

efiring at manini:~/temp/nnbf$   python test_nnbf.py
loading data... this could take a while
testing nnbf...
     10 trials: mean=0.0150, min=0.0100
testing numpy...
     10 trials: mean=0.0660, min=0.0600

It is all a matter of keeping Python objects and function calls out of 
inner loops.  I suspect there is quite a bit more that could be done in 
that regard, but I haven't looked.

Eric
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: nnbf.pyx
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090107/9d163cb6/attachment.ksh>

From ezindy at gmail.com  Wed Jan  7 20:13:15 2009
From: ezindy at gmail.com (Egor Zindy)
Date: Thu, 8 Jan 2009 10:13:15 +0900
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
	<e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
	<a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>
Message-ID: <c156de00901071713i31876ec3u84d39b2ac4f27ab3@mail.gmail.com>

Hello Rich,

This is very strange. I got to test my example again, as long as you don't
do a
%include "dftmagnitude.h"
somewhere in the dftmagnitude.i, it's perfectly possible to do a
%rename (sms_spectrumMag) my_spectrumMag;
(see dftmagnitude3.zip attached in my previous mail and this one).

So things for you to check:
  * does the simple dftmagnitude3.zip compile on your system?
  * what version of SWIG are you using? (I used 1.3.36 provided with cygwin)
  * do you have a %include statement somewhere in your own .i file?

Matthieu, if you read this, there's a complete example provided in
dftmagnitude3.zip.
  * Wrapped function sms_spectrumMag in dftmagnitude.c and .h
  * SWIG wrapper dftmagnitude.i uses %inline and %rename statements
  * Example uses a modified numpy.i (see the previous mails in the thread).
  * test example provided in test_dftmagnitude.py

Haven't tested it under Linux, but under winxp/cygwin/mingw32, the following
works for me (in cygwin):

$ python setup_dftmagnitude.py build -cmingw32 ; mv
build/lib.win32-2.5/_dftmagnitude.pyd .
$ python test_dftmagnitude.py

Regards,
Egor

--
My Python:
$ python -i
Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
on win32

My SWIG:
$ swig -version

SWIG Version 1.3.36

Compiled with g++ [i686-pc-cygwin]
Please see http://www.swig.org for reporting bugs and further information




On Thu, Jan 8, 2009 at 1:43 AM, Rich E <reakinator at gmail.com> wrote:

> Here is my example, trying to wrap the function sms_spectrumMag that
> we have been dealing with:
>
> %apply (int DIM1, float* IN_ARRAY1) {(int sizeInArray, float* pInArray)};
> %apply (int DIM1, float* INPLACE_ARRAY1) {(int sizeOutArray, float*
> pOutArray)};
>
> %inline %{
>
>        void my_spectrumMag( int sizeInArray, float *pInArray, int
> sizeOutArray, float *pOutArray)
>        {
>                sms_spectrumMag(sizeOutArray, pInArray, pOutArray);
>        }
>
> %}
>
>
> at this point,  have the new function my_spectrumMag that wraps
> sms_spectrumMag() and provides arguments that can be typemapped using
> numpy.i  Now, I don't want to have to call the function
> my_spectrumMag() in python, I want to use the original name, I would
> like to call the function as:
>
> sms_spectrumMag(numpyArray1, numpyArray2)
>
> But, trying to %rename my_spectrumMag to sms_spectrumMag does not
> work, the original sms_spectrumMag gets called in python instead.
> Trying to %ignore the original function first as follows removes the
> sms_spectrumMag completely from the module and I am left with
> my_spectrumMag:
>
> %ignore sms_spectrumMag;
> %rename (sms_spectrumMag) my_spectrumMag;
>
>
> Do you see my problem?
>
>
> On Wed, Jan 7, 2009 at 8:58 AM, Matthieu Brucher
> <matthieu.brucher at gmail.com> wrote:
> > 2009/1/6 Rich E <reakinator at gmail.com>:
> >> This helped immensely.  I feel like I am getting close to being able
> >> to accomplish what I would like with SWIG: producing a python module
> >> that can be very 'python-like', while co-existing with the c library
> >> that is very 'c-like'.
> >>
> >> There is one question still remaining though, is it possible to make
> >> the wrapped function have the same name still?  Using either
> >> my_spectrumMag or spectrumMag means I have to create a number of
> >> inconsistencies between the python module and the c library.  It is
> >> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
> >> wrapped one, with the same name.  But my attempts at doing this so far
> >> have not compiled because of name conflictions.
> >
> > Ok course you can. The function is renamed only if you say so. Perhaps
> > can you provide a small example of what doesn't work at the moment ?
> >
> >> Thanks for the help, I think you are doing great things with this
> >> numpy interface/typemaps system.
> >
> > Matthieu
> > --
> > Information System Engineer, Ph.D.
> > Website: http://matthieu-brucher.developpez.com/
> > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090108/61800800/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dftmagnitude3.zip
Type: application/zip
Size: 10887 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090108/61800800/attachment.zip>

From bevan07 at gmail.com  Wed Jan  7 22:02:50 2009
From: bevan07 at gmail.com (Bevan Jenkins)
Date: Thu, 8 Jan 2009 03:02:50 +0000 (UTC)
Subject: [Numpy-discussion] Accumulate values that are below threshold
Message-ID: <loom.20090108T024310-329@post.gmane.org>

Hello,

Sometimes the hardest part of a problem is articulating it.  Hopefully I can
describe what I am trying to do - at least enough to get some help.

I am trying to compare values to a threshold and when the values are lower than
the threshold they are added to the value in my set until the threshold is
reached.  Everytime the threshold is reached I want the index and value
(accumulated).

Hopefully the example below will help

threshold =1.0
for indx,val in enumerate(Q):
    print indx,val

0 100.0
1 20.0
2 16.0
3 7.0
4 3.0
5 1.5
6 0.8
7 0.6
8 0.5
9 0.2
10 0.2
11 0.1
12 0.1

The output I would like is (number of elements and value)
0 100.0
1 20.0
2 16.0
3 7.0
4 3.0
5 1.5
7 1.4
11 1.0


The 1st 6 elements are easy as they are all greater than or equal to the
threshold(1.0).  Once the values drop below the threshold the next value is
added until the threshold is reached.


Any help is appreciated,
Bevan Jenkins



From stefan at sun.ac.za  Thu Jan  8 01:36:41 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 8 Jan 2009 08:36:41 +0200
Subject: [Numpy-discussion] Accumulate values that are below threshold
In-Reply-To: <loom.20090108T024310-329@post.gmane.org>
References: <loom.20090108T024310-329@post.gmane.org>
Message-ID: <9457e7c80901072236ua4d0667j66fe0ed96f9fccdb@mail.gmail.com>

Hi Bevan

Since the number of output elements are unknown, I don't think you can
implement this efficiently using arrays.  If your dataset isn't too
large, a for-loop should do the trick.  Otherwise, you may have to run
your code through Cython, which optimises for-loops around Python
lists.

thresh = 1.0
carry = 0
output = []
for idx, val in data:
    carry += val
    if (carry - thresh) >= -1e-15:
        output.append((idx, carry))
        carry = 0

The comparison line above, "(carry - thresh0 >= -1e-15", may look
strange -- it basically just does "carry >= thresh".  For some reason
I don't quite understand, when accumulating floats, it sometimes
happens that "1.0 != 1.0", so I use 1e-15 as protection.

Regards
St?fan

2009/1/8 Bevan Jenkins <bevan07 at gmail.com>:
> Hello,
>
> Sometimes the hardest part of a problem is articulating it.  Hopefully I can
> describe what I am trying to do - at least enough to get some help.
>
> I am trying to compare values to a threshold and when the values are lower than
> the threshold they are added to the value in my set until the threshold is
> reached.  Everytime the threshold is reached I want the index and value
> (accumulated).
>
> Hopefully the example below will help
>
> threshold =1.0
> for indx,val in enumerate(Q):
>    print indx,val
>
> 0 100.0
> 1 20.0
> 2 16.0
> 3 7.0
> 4 3.0
> 5 1.5
> 6 0.8
> 7 0.6
> 8 0.5
> 9 0.2
> 10 0.2
> 11 0.1
> 12 0.1
>
> The output I would like is (number of elements and value)
> 0 100.0
> 1 20.0
> 2 16.0
> 3 7.0
> 4 3.0
> 5 1.5
> 7 1.4
> 11 1.0
>
>
> The 1st 6 elements are easy as they are all greater than or equal to the
> threshold(1.0).  Once the values drop below the threshold the next value is
> added until the threshold is reached.
>
>
> Any help is appreciated,
> Bevan Jenkins


From ezindy at gmail.com  Thu Jan  8 11:22:19 2009
From: ezindy at gmail.com (Egor Zindy)
Date: Fri, 9 Jan 2009 01:22:19 +0900
Subject: [Numpy-discussion] ANN: numpy.i - added managed deallocation to
	ARGOUTVIEW_ARRAY1 (ARGOUTVIEWM_ARRAY1)
In-Reply-To: <493D1926.4010506@gmail.com>
References: <491F8F4A.30009@gmail.com> <49231156.1060209@gmail.com>
	<49231EB3.8060802@noaa.gov> <492538B9.10202@gmail.com>
	<493D1926.4010506@gmail.com>
Message-ID: <c156de00901080822r4dc81d96tee1f09e90752fbe3@mail.gmail.com>

Hello list,

I've moved my wiki to the scipy cookbook:
http://www.scipy.org/Cookbook/SWIG_Memory_Deallocation

For the time being, the listed example files are still stored on the google
code SVN, but these could easily be moved if necessary.

I've also just finished adding an ARGOUTVIEWM_ARRAY2 example. The example
shows how to return a two-dimensional array from C which also benefits from
the automatic memory deallocation.

A naive "crop" function is wrapped using SWIG/numpy.i and returns a slice of
the input array. When used as

array_out = crop.crop(array_in, d1_0,d1_1, d2_0,d2_1)

it is equivalent to the native numpy slicing

array_out = array_in[d1_0:d1_1, d2_0:d2_1]


Hope this helps!

Regards,
Egor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090109/5dd052f9/attachment.html>

From chanley at stsci.edu  Thu Jan  8 11:37:51 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 08 Jan 2009 11:37:51 -0500
Subject: [Numpy-discussion] change made to test_print.py
Message-ID: <49662BDF.7000806@stsci.edu>

Hi,

I've committed the following change to test_print.py to fix one of the 
tests.

Index: test_print.py
===================================================================
--- test_print.py       (revision 6302)
+++ test_print.py       (working copy)
@@ -154,7 +154,7 @@
          else:
              locale.setlocale(locale.LC_NUMERIC, 'FRENCH')

-        assert_equal(str(tp(1.2)), str(float(1.2)),
+        assert_equal(locale.format("%f",tp(1.2)), 
locale.format("%f",float(1.2)),
                       err_msg='Failed locale test for type %s' % tp)
      finally:
          locale.setlocale(locale.LC_NUMERIC, locale=curloc)


Chris

-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From jdh2358 at gmail.com  Thu Jan  8 12:32:32 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 8 Jan 2009 11:32:32 -0600
Subject: [Numpy-discussion] my cython is slow
In-Reply-To: <49653CD4.90808@hawaii.edu>
References: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>
	<49653CD4.90808@hawaii.edu>
Message-ID: <88e473830901080932i72003322idbaedb9af4de1dae@mail.gmail.com>

On Wed, Jan 7, 2009 at 5:37 PM, Eric Firing <efiring at hawaii.edu> wrote:

> A couple small changes speed it up quite a bit:
>
> efiring at manini:~/temp/nnbf$   python test_nnbf.py
> loading data... this could take a while
> testing nnbf...
>    10 trials: mean=0.0150, min=0.0100
> testing numpy...
>    10 trials: mean=0.0660, min=0.0600
>
> It is all a matter of keeping Python objects and function calls out of inner
> loops.  I suspect there is quite a bit more that could be done in that
> regard, but I haven't looked.

Much  faster, but no longer correct, as you'll see if you uncomment
out the nose test test_neighbors that compare actual vs desired.

Is the pointer arithmetic correct:

  dataptr + i

I would have thought perhaps:

  dataptr + i*n

but this is segfaulting.  Do we need to use a stride?

JDH


From efiring at hawaii.edu  Thu Jan  8 13:34:14 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Thu, 08 Jan 2009 08:34:14 -1000
Subject: [Numpy-discussion] my cython is slow
In-Reply-To: <88e473830901080932i72003322idbaedb9af4de1dae@mail.gmail.com>
References: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>
	<49653CD4.90808@hawaii.edu>
	<88e473830901080932i72003322idbaedb9af4de1dae@mail.gmail.com>
Message-ID: <49664726.7020309@hawaii.edu>

John Hunter wrote:
> On Wed, Jan 7, 2009 at 5:37 PM, Eric Firing <efiring at hawaii.edu> wrote:
> 
>> A couple small changes speed it up quite a bit:
>>
>> efiring at manini:~/temp/nnbf$   python test_nnbf.py
>> loading data... this could take a while
>> testing nnbf...
>>    10 trials: mean=0.0150, min=0.0100
>> testing numpy...
>>    10 trials: mean=0.0660, min=0.0600
>>
>> It is all a matter of keeping Python objects and function calls out of inner
>> loops.  I suspect there is quite a bit more that could be done in that
>> regard, but I haven't looked.
> 
> Much  faster, but no longer correct, as you'll see if you uncomment
> out the nose test test_neighbors that compare actual vs desired.
> 
> Is the pointer arithmetic correct:
> 
>   dataptr + i
> 
> I would have thought perhaps:
> 
>   dataptr + i*n
> 
> but this is segfaulting.  Do we need to use a stride?

Sorry, I was too hasty.  Yes, it seems like i*n should be correct, but 
it isn't; we are missing something simple and fundamental here.  I don't 
see it immediately, and won't be able to look at it for a while.

Eric

> 
> JDH
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From cournape at gmail.com  Thu Jan  8 14:15:09 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 9 Jan 2009 04:15:09 +0900
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <49662BDF.7000806@stsci.edu>
References: <49662BDF.7000806@stsci.edu>
Message-ID: <5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>

On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley at stsci.edu> wrote:
> Hi,
>
> I've committed the following change to test_print.py to fix one of the
> tests.
>

Hi Christopher,

Please do not modify those tests - they are supposed to fail,

David


From chanley at stsci.edu  Thu Jan  8 14:29:11 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 08 Jan 2009 14:29:11 -0500
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
Message-ID: <49665407.5030505@stsci.edu>

David Cournapeau wrote:
> On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>> Hi,
>>
>> I've committed the following change to test_print.py to fix one of the
>> tests.
>>
> 
> Hi Christopher,
> 
> Please do not modify those tests - they are supposed to fail,
> 
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

Hi David,

Sorry.  Should these tests be generating a "known failures" then?

======================================================================
FAIL: test_print.test_locale_single
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/Users/chanley/dev/site-packages/lib/python/nose/case.py", line 
182, in runTest
     self.test(*self.arg)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/testing/decorators.py", 
line 82, in skipper
     return f(*args, **kwargs)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", 
line 165, in test_locale_single
     return _test_locale_independance(np.float32)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", 
line 158, in _test_locale_independance
     err_msg='Failed locale test for type %s' % tp)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/testing/utils.py", 
line 183, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal: Failed locale test for type <type 'numpy.float32'>
  ACTUAL: '1,2'
  DESIRED: '1.2'

======================================================================
FAIL: test_print.test_locale_double
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/Users/chanley/dev/site-packages/lib/python/nose/case.py", line 
182, in runTest
     self.test(*self.arg)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/testing/decorators.py", 
line 82, in skipper
     return f(*args, **kwargs)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", 
line 170, in test_locale_double
     return _test_locale_independance(np.double)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", 
line 158, in _test_locale_independance
     err_msg='Failed locale test for type %s' % tp)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/testing/utils.py", 
line 183, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal: Failed locale test for type <type 'numpy.float64'>
  ACTUAL: '1,2'
  DESIRED: '1.2'

======================================================================
FAIL: test_print.test_locale_longdouble
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/Users/chanley/dev/site-packages/lib/python/nose/case.py", line 
182, in runTest
     self.test(*self.arg)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/testing/decorators.py", 
line 82, in skipper
     return f(*args, **kwargs)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", 
line 175, in test_locale_longdouble
     return _test_locale_independance(np.longdouble)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", 
line 158, in _test_locale_independance
     err_msg='Failed locale test for type %s' % tp)
   File 
"/Users/chanley/dev/site-packages/lib/python/numpy/testing/utils.py", 
line 183, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal: Failed locale test for type <type 'numpy.float128'>
  ACTUAL: '1,2'
  DESIRED: '1.2'

Thanks,
Chris


-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From cournape at gmail.com  Thu Jan  8 14:48:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 9 Jan 2009 04:48:02 +0900
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <49665407.5030505@stsci.edu>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
Message-ID: <5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>

On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley at stsci.edu> wrote:
> David Cournapeau wrote:
>> On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>> Hi,
>>>
>>> I've committed the following change to test_print.py to fix one of the
>>> tests.
>>>
>>
>> Hi Christopher,
>>
>> Please do not modify those tests - they are supposed to fail,
>>
>> David
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> Hi David,
>
> Sorry.  Should these tests be generating a "known failures" then?

No. The problem are known, and are being fixed (in a branch). Since
the problem is only in the development trunk, I don't see any problem
with having failures for some time,

David


From jdh2358 at gmail.com  Thu Jan  8 14:56:02 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 8 Jan 2009 13:56:02 -0600
Subject: [Numpy-discussion] my cython is slow
In-Reply-To: <49664726.7020309@hawaii.edu>
References: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>
	<49653CD4.90808@hawaii.edu>
	<88e473830901080932i72003322idbaedb9af4de1dae@mail.gmail.com>
	<49664726.7020309@hawaii.edu>
Message-ID: <88e473830901081156s467057d8u8dddccaae9b227dd@mail.gmail.com>

On Thu, Jan 8, 2009 at 12:34 PM, Eric Firing <efiring at hawaii.edu> wrote:
> John Hunter wrote:
>> On Wed, Jan 7, 2009 at 5:37 PM, Eric Firing <efiring at hawaii.edu> wrote:
>>
>>> A couple small changes speed it up quite a bit:
>>>
>>> efiring at manini:~/temp/nnbf$   python test_nnbf.py
>>> loading data... this could take a while
>>> testing nnbf...
>>>    10 trials: mean=0.0150, min=0.0100
>>> testing numpy...
>>>    10 trials: mean=0.0660, min=0.0600
>>>
>>> It is all a matter of keeping Python objects and function calls out of inner
>>> loops.  I suspect there is quite a bit more that could be done in that
>>> regard, but I haven't looked.
>>
>> Much  faster, but no longer correct, as you'll see if you uncomment
>> out the nose test test_neighbors that compare actual vs desired.
>>
>> Is the pointer arithmetic correct:
>>
>>   dataptr + i
>>
>> I would have thought perhaps:
>>
>>   dataptr + i*n
>>
>> but this is segfaulting.  Do we need to use a stride?
>
> Sorry, I was too hasty.  Yes, it seems like i*n should be correct, but
> it isn't; we are missing something simple and fundamental here.  I don't
> see it immediately, and won't be able to look at it for a while.

OK, the code at

  > svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/trunk/py4science/examples/pyrex/nnbf

now passes the correctness tests and is approx ten time faster than
the numpy version.

I borrowed the "raw_data" idiom from Anne's  ckdtree.pyx, though I
don't really understand why it is different that what we were doing
with the data.data ptr.  I am also not sure if I need to be doing any
memory management when I resize the buffer and reset raw_data....

"""
A brute force nearest neighbor routine with incremental add.  The
internal array data structure grows as you add points
"""

import numpy as np
cimport numpy as np

cdef extern from "math.h":
     float sqrt(float)

cdef inline int is_neighbor(int n, double*row, double*pp, double d2max):
    """
    return 1 if the sum-of-squares of n length array row[j]-pp[j] <= d2max
    """
    cdef int j
    cdef double d, d2

    d2 = 0.

    for j in range(n):
        d = row[j] - pp[j]
        d2 += d*d
        if d2>d2max:
            return 0
    return 1

cdef class NNBF:
    cdef readonly object data
    cdef double* raw_data
    cdef readonly int n, numrows, numpoints

    def __init__(self, n):
        """
        create a buffer to hold n dimensional points
        """
        cdef np.ndarray[double, ndim=2] inner_data


        self.n = n
        self.numrows = 10000
        #  XXX how to create empty as contiguous w/o copy?
        data = np.empty((self.numrows, self.n), dtype=np.float)
        self.data = np.ascontiguousarray(data, dtype=np.float)
        inner_data = self.data
        self.raw_data = <double*>inner_data.data
        self.numpoints = 0


    def add(NNBF self, object point):
        """
        add a point to the buffer, grow if necessary
        """
        cdef np.ndarray[double, ndim=2] inner_data
        cdef np.ndarray[double, ndim=1] pp
        pp = np.array(point).astype(np.float)


        self.data[self.numpoints] = pp
        self.numpoints += 1
        if self.numpoints==self.numrows:
            ## XXX do I need to do memory management here, eg free
            ## raw_data if I were using it?
            self.numrows *= 2
            newdata = np.empty((self.numrows, self.n), np.float)
            newdata[:self.numpoints] = self.data
            self.data = np.ascontiguousarray(newdata, dtype=np.float)
            inner_data = self.data
            self.raw_data = <double*>inner_data.data


    def get_data(NNBF self):
        """
        return a copy of data added so far as a numpoints x n array
        """
        return self.data[:self.numpoints]


    def find_neighbors(NNBF self, object point, double radius):
        """
        return a list of indices into data which are within radius
        from point
        """
        cdef int i, neighbor, n
        cdef double d2max
        cdef np.ndarray[double, ndim=1] pp

        # avoid python array indexing in the inner loop
        if len(point)!=self.n:
            raise ValueError('Expected a length %d vector'%self.n)

        pp = np.asarray(point).astype(np.float)

        d2max = radius*radius
        neighbors = []

        # don't do a python lookup inside the loop
        n = self.n

        for i in range(self.numpoints):
            neighbor = is_neighbor(
                n,
                self.raw_data + i*n,
                <double*>pp.data,
                d2max)

            # if the number of points in the cluster is small, the
            # python list performance should not kill us
            if neighbor:
                neighbors.append(i)

        return neighbors

    def find_neighbors_numpy(self, point, radius):
        """
        do a plain ol numpy lookup to compare performance and output

          *data* is a numpoints x numdims array
          *point* is a numdims length vector
          radius is the max distance distance

        return an array of indices into data which are within radius
        """
        data = self.get_data()
        distance = data - point
        r = np.sqrt((distance*distance).sum(axis=1))
        return np.nonzero(r<=radius)[0]


From chanley at stsci.edu  Thu Jan  8 15:11:55 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 08 Jan 2009 15:11:55 -0500
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
Message-ID: <49665E0B.3030009@stsci.edu>

David Cournapeau wrote:
> On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>> David Cournapeau wrote:
>>> On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>>> Hi,
>>>>
>>>> I've committed the following change to test_print.py to fix one of the
>>>> tests.
>>>>
>>> Hi Christopher,
>>>
>>> Please do not modify those tests - they are supposed to fail,
>>>
>>> David
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> Hi David,
>>
>> Sorry.  Should these tests be generating a "known failures" then?
> 
> No. The problem are known, and are being fixed (in a branch). Since
> the problem is only in the development trunk, I don't see any problem
> with having failures for some time,
> 
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

I would disagree.  If you were to attempt the following:

n = numpy.test()
n.wasSuccessful()

You expect the result to be 'True'.  If not it is necessary to find out 
why.  Right now the following occurs:

 >>> n.wasSuccessful()
False

I have no way of knowing that you wanted those tests to fail unless you 
have them marked as KNOWNFAIL. Since we use numpy in our production 
systems I need to determine why numpy is failing.  We track the changes 
on the trunk because we need to know how changes will effect our code 
prior to our customers downloading the latest numpy release.  This 
'False' return value from wasSuccessful() means that our automated 
systems tell us that numpy is broken.

The common assumption is that tests are not suppose to fail.  If they 
are failing then there is a problem.

If you have tests that you want to fail, either make the failure a 
condition of the test passing or move the tests to the branch where you 
are doing your development.

Thanks,
Chris



-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From dagss at student.matnat.uio.no  Thu Jan  8 15:30:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 Jan 2009 21:30:24 +0100
Subject: [Numpy-discussion] my cython is slow
In-Reply-To: <88e473830901081156s467057d8u8dddccaae9b227dd@mail.gmail.com>
References: <88e473830901071452g12305ec4r8d3deb52547bcc71@mail.gmail.com>	<49653CD4.90808@hawaii.edu>	<88e473830901080932i72003322idbaedb9af4de1dae@mail.gmail.com>	<49664726.7020309@hawaii.edu>
	<88e473830901081156s467057d8u8dddccaae9b227dd@mail.gmail.com>
Message-ID: <49666260.3070408@student.matnat.uio.no>

Some of the problems you encounter could probably be remedied by better 
support in Cython for some situations. I've filed two feature request 
tickets for myself, but I have no idea when or if I'll get around to them.

http://trac.cython.org/cython_trac/ticket/177
http://trac.cython.org/cython_trac/ticket/178

Dag Sverre

John Hunter wrote:
> On Thu, Jan 8, 2009 at 12:34 PM, Eric Firing <efiring at hawaii.edu> wrote:
>> John Hunter wrote:
>>> On Wed, Jan 7, 2009 at 5:37 PM, Eric Firing <efiring at hawaii.edu> wrote:
>>>
>>>> A couple small changes speed it up quite a bit:
>>>>
>>>> efiring at manini:~/temp/nnbf$   python test_nnbf.py
>>>> loading data... this could take a while
>>>> testing nnbf...
>>>>    10 trials: mean=0.0150, min=0.0100
>>>> testing numpy...
>>>>    10 trials: mean=0.0660, min=0.0600
>>>>
>>>> It is all a matter of keeping Python objects and function calls out of inner
>>>> loops.  I suspect there is quite a bit more that could be done in that
>>>> regard, but I haven't looked.
>>> Much  faster, but no longer correct, as you'll see if you uncomment
>>> out the nose test test_neighbors that compare actual vs desired.
>>>
>>> Is the pointer arithmetic correct:
>>>
>>>   dataptr + i
>>>
>>> I would have thought perhaps:
>>>
>>>   dataptr + i*n
>>>
>>> but this is segfaulting.  Do we need to use a stride?
>> Sorry, I was too hasty.  Yes, it seems like i*n should be correct, but
>> it isn't; we are missing something simple and fundamental here.  I don't
>> see it immediately, and won't be able to look at it for a while.
> 
> OK, the code at
> 
>   > svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/trunk/py4science/examples/pyrex/nnbf
> 
> now passes the correctness tests and is approx ten time faster than
> the numpy version.
> 
> I borrowed the "raw_data" idiom from Anne's  ckdtree.pyx, though I
> don't really understand why it is different that what we were doing
> with the data.data ptr.  I am also not sure if I need to be doing any
> memory management when I resize the buffer and reset raw_data....
> 
> """
> A brute force nearest neighbor routine with incremental add.  The
> internal array data structure grows as you add points
> """
> 
> import numpy as np
> cimport numpy as np
> 
> cdef extern from "math.h":
>      float sqrt(float)
> 
> cdef inline int is_neighbor(int n, double*row, double*pp, double d2max):
>     """
>     return 1 if the sum-of-squares of n length array row[j]-pp[j] <= d2max
>     """
>     cdef int j
>     cdef double d, d2
> 
>     d2 = 0.
> 
>     for j in range(n):
>         d = row[j] - pp[j]
>         d2 += d*d
>         if d2>d2max:
>             return 0
>     return 1
> 
> cdef class NNBF:
>     cdef readonly object data
>     cdef double* raw_data
>     cdef readonly int n, numrows, numpoints
> 
>     def __init__(self, n):
>         """
>         create a buffer to hold n dimensional points
>         """
>         cdef np.ndarray[double, ndim=2] inner_data
> 
> 
>         self.n = n
>         self.numrows = 10000
>         #  XXX how to create empty as contiguous w/o copy?
>         data = np.empty((self.numrows, self.n), dtype=np.float)
>         self.data = np.ascontiguousarray(data, dtype=np.float)
>         inner_data = self.data
>         self.raw_data = <double*>inner_data.data
>         self.numpoints = 0
> 
> 
>     def add(NNBF self, object point):
>         """
>         add a point to the buffer, grow if necessary
>         """
>         cdef np.ndarray[double, ndim=2] inner_data
>         cdef np.ndarray[double, ndim=1] pp
>         pp = np.array(point).astype(np.float)
> 
> 
>         self.data[self.numpoints] = pp
>         self.numpoints += 1
>         if self.numpoints==self.numrows:
>             ## XXX do I need to do memory management here, eg free
>             ## raw_data if I were using it?
>             self.numrows *= 2
>             newdata = np.empty((self.numrows, self.n), np.float)
>             newdata[:self.numpoints] = self.data
>             self.data = np.ascontiguousarray(newdata, dtype=np.float)
>             inner_data = self.data
>             self.raw_data = <double*>inner_data.data
> 
> 
>     def get_data(NNBF self):
>         """
>         return a copy of data added so far as a numpoints x n array
>         """
>         return self.data[:self.numpoints]
> 
> 
>     def find_neighbors(NNBF self, object point, double radius):
>         """
>         return a list of indices into data which are within radius
>         from point
>         """
>         cdef int i, neighbor, n
>         cdef double d2max
>         cdef np.ndarray[double, ndim=1] pp
> 
>         # avoid python array indexing in the inner loop
>         if len(point)!=self.n:
>             raise ValueError('Expected a length %d vector'%self.n)
> 
>         pp = np.asarray(point).astype(np.float)
> 
>         d2max = radius*radius
>         neighbors = []
> 
>         # don't do a python lookup inside the loop
>         n = self.n
> 
>         for i in range(self.numpoints):
>             neighbor = is_neighbor(
>                 n,
>                 self.raw_data + i*n,
>                 <double*>pp.data,
>                 d2max)
> 
>             # if the number of points in the cluster is small, the
>             # python list performance should not kill us
>             if neighbor:
>                 neighbors.append(i)
> 
>         return neighbors
> 
>     def find_neighbors_numpy(self, point, radius):
>         """
>         do a plain ol numpy lookup to compare performance and output
> 
>           *data* is a numpoints x numdims array
>           *point* is a numdims length vector
>           radius is the max distance distance
> 
>         return an array of indices into data which are within radius
>         """
>         data = self.get_data()
>         distance = data - point
>         r = np.sqrt((distance*distance).sum(axis=1))
>         return np.nonzero(r<=radius)[0]
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


-- 
Dag Sverre


From cournape at gmail.com  Thu Jan  8 15:26:17 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 9 Jan 2009 05:26:17 +0900
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <49665E0B.3030009@stsci.edu>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
Message-ID: <5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>

On Fri, Jan 9, 2009 at 5:11 AM, Christopher Hanley <chanley at stsci.edu> wrote:
> David Cournapeau wrote:
>> On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>> David Cournapeau wrote:
>>>> On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>>>> Hi,
>>>>>
>>>>> I've committed the following change to test_print.py to fix one of the
>>>>> tests.
>>>>>
>>>> Hi Christopher,
>>>>
>>>> Please do not modify those tests - they are supposed to fail,
>>>>
>>>> David
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Numpy-discussion at scipy.org
>>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>> Hi David,
>>>
>>> Sorry.  Should these tests be generating a "known failures" then?
>>
>> No. The problem are known, and are being fixed (in a branch). Since
>> the problem is only in the development trunk, I don't see any problem
>> with having failures for some time,
>>
>> David
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> I would disagree.  If you were to attempt the following:
>
> n = numpy.test()
> n.wasSuccessful()
>
> You expect the result to be 'True'.  If not it is necessary to find out
> why.  Right now the following occurs:
>
>  >>> n.wasSuccessful()
> False
>
> I have no way of knowing that you wanted those tests to fail unless you
> have them marked as KNOWNFAIL. Since we use numpy in our production
> systems I need to determine why numpy is failing.  We track the changes
> on the trunk because we need to know how changes will effect our code
> prior to our customers downloading the latest numpy release.

I don't understand: you can't expect the trunk to always work. We try
not to break it - but sometimes it does not work.

Personally, I don't like knownfailure much anyway: I feel like it is
too easy to tag one test known failure, and then nobody cares about it
anymore. Those formatting problems were already problems before - the
tests only show the problem, it does not cause the problem, so I don't
understand why it is so important: a 100 % running test suite with a
problem which is not shown or a 95 % running test suite with the
problem is the same thing; the code in numpy itself is exactly the
same.

David


From robert.kern at gmail.com  Thu Jan  8 15:32:51 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 8 Jan 2009 15:32:51 -0500
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
Message-ID: <3d375d730901081232m70d17347k16301fe8daf99591@mail.gmail.com>

On Thu, Jan 8, 2009 at 15:26, David Cournapeau <cournape at gmail.com> wrote:

> Personally, I don't like knownfailure much anyway: I feel like it is
> too easy to tag one test known failure, and then nobody cares about it
> anymore. Those formatting problems were already problems before - the
> tests only show the problem, it does not cause the problem, so I don't
> understand why it is so important: a 100 % running test suite with a
> problem which is not shown or a 95 % running test suite with the
> problem is the same thing; the code in numpy itself is exactly the
> same.

Don't check in failing tests without using knownfailure. Unit tests
are used by others to determine whether or not *they* broke things or
whether their installation failed. By checking in a failing test, you
are sending others on a wild goose chase trying to figure out what
they did wrong when they didn't.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Jan  8 15:33:28 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 8 Jan 2009 15:33:28 -0500
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <49662BDF.7000806@stsci.edu>
References: <49662BDF.7000806@stsci.edu>
Message-ID: <3d375d730901081233j57897718i91f17828cb73834f@mail.gmail.com>

On Thu, Jan 8, 2009 at 11:37, Christopher Hanley <chanley at stsci.edu> wrote:
> Hi,
>
> I've committed the following change to test_print.py to fix one of the
> tests.
>
> Index: test_print.py
> ===================================================================
> --- test_print.py       (revision 6302)
> +++ test_print.py       (working copy)
> @@ -154,7 +154,7 @@
>          else:
>              locale.setlocale(locale.LC_NUMERIC, 'FRENCH')
>
> -        assert_equal(str(tp(1.2)), str(float(1.2)),
> +        assert_equal(locale.format("%f",tp(1.2)),
> locale.format("%f",float(1.2)),

Note that this does not test anything. %f coerces the input to a
Python float anyways.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From chanley at stsci.edu  Thu Jan  8 15:36:37 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 08 Jan 2009 15:36:37 -0500
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>	<49665407.5030505@stsci.edu>	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
Message-ID: <496663D5.4090005@stsci.edu>

David Cournapeau wrote:
> On Fri, Jan 9, 2009 at 5:11 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>> David Cournapeau wrote:
>>> On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>>> David Cournapeau wrote:
>>>>> On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've committed the following change to test_print.py to fix one of the
>>>>>> tests.
>>>>>>
>>>>> Hi Christopher,
>>>>>
>>>>> Please do not modify those tests - they are supposed to fail,
>>>>>
>>>>> David
>>>>> _______________________________________________
>>>>> Numpy-discussion mailing list
>>>>> Numpy-discussion at scipy.org
>>>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>> Hi David,
>>>>
>>>> Sorry.  Should these tests be generating a "known failures" then?
>>> No. The problem are known, and are being fixed (in a branch). Since
>>> the problem is only in the development trunk, I don't see any problem
>>> with having failures for some time,
>>>
>>> David
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> I would disagree.  If you were to attempt the following:
>>
>> n = numpy.test()
>> n.wasSuccessful()
>>
>> You expect the result to be 'True'.  If not it is necessary to find out
>> why.  Right now the following occurs:
>>
>>  >>> n.wasSuccessful()
>> False
>>
>> I have no way of knowing that you wanted those tests to fail unless you
>> have them marked as KNOWNFAIL. Since we use numpy in our production
>> systems I need to determine why numpy is failing.  We track the changes
>> on the trunk because we need to know how changes will effect our code
>> prior to our customers downloading the latest numpy release.
> 
> I don't understand: you can't expect the trunk to always work. We try
> not to break it - but sometimes it does not work.
> 
> Personally, I don't like knownfailure much anyway: I feel like it is
> too easy to tag one test known failure, and then nobody cares about it
> anymore. Those formatting problems were already problems before - the
> tests only show the problem, it does not cause the problem, so I don't
> understand why it is so important: a 100 % running test suite with a
> problem which is not shown or a 95 % running test suite with the
> problem is the same thing; the code in numpy itself is exactly the
> same.
> 
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

I do not expect the trunk to always work.  I even expect it to have 
bugs.  However, I do not expect there to be test failures for known 
reasons that result in wasSuccessful() returning false.  This is a bad 
programming practice.  It creates work for people trying to figure out 
what is wrong when the answer is already know.

Chris


-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From reakinator at gmail.com  Thu Jan  8 15:43:35 2009
From: reakinator at gmail.com (Rich E)
Date: Thu, 8 Jan 2009 21:43:35 +0100
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <c156de00901071713i31876ec3u84d39b2ac4f27ab3@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
	<e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
	<a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>
	<c156de00901071713i31876ec3u84d39b2ac4f27ab3@mail.gmail.com>
Message-ID: <a3297bc90901081243g3c04e089i94417f7d3438d3a5@mail.gmail.com>

I am using %includ "sms.h", which is what is wrapping all my
functions.  Without doing this, I have to hand-wrap every function in
the header file!

Is there a way to exclude certain definitions from my c header file
when using %include, so that I can hand wrap them instead?

On Thu, Jan 8, 2009 at 2:13 AM, Egor Zindy <ezindy at gmail.com> wrote:
> Hello Rich,
>
> This is very strange. I got to test my example again, as long as you don't
> do a
> %include "dftmagnitude.h"
> somewhere in the dftmagnitude.i, it's perfectly possible to do a
> %rename (sms_spectrumMag) my_spectrumMag;
> (see dftmagnitude3.zip attached in my previous mail and this one).
>
> So things for you to check:
>   * does the simple dftmagnitude3.zip compile on your system?
>   * what version of SWIG are you using? (I used 1.3.36 provided with cygwin)
>   * do you have a %include statement somewhere in your own .i file?
>
> Matthieu, if you read this, there's a complete example provided in
> dftmagnitude3.zip.
>   * Wrapped function sms_spectrumMag in dftmagnitude.c and .h
>   * SWIG wrapper dftmagnitude.i uses %inline and %rename statements
>   * Example uses a modified numpy.i (see the previous mails in the thread).
>   * test example provided in test_dftmagnitude.py
>
> Haven't tested it under Linux, but under winxp/cygwin/mingw32, the following
> works for me (in cygwin):
>
> $ python setup_dftmagnitude.py build -cmingw32 ; mv
> build/lib.win32-2.5/_dftmagnitude.pyd .
> $ python test_dftmagnitude.py
>
> Regards,
> Egor
>
> --
> My Python:
> $ python -i
> Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
> on win32
>
> My SWIG:
> $ swig -version
>
> SWIG Version 1.3.36
>
> Compiled with g++ [i686-pc-cygwin]
> Please see http://www.swig.org for reporting bugs and further information
>
>
>
>
> On Thu, Jan 8, 2009 at 1:43 AM, Rich E <reakinator at gmail.com> wrote:
>>
>> Here is my example, trying to wrap the function sms_spectrumMag that
>> we have been dealing with:
>>
>> %apply (int DIM1, float* IN_ARRAY1) {(int sizeInArray, float* pInArray)};
>> %apply (int DIM1, float* INPLACE_ARRAY1) {(int sizeOutArray, float*
>> pOutArray)};
>>
>> %inline %{
>>
>>        void my_spectrumMag( int sizeInArray, float *pInArray, int
>> sizeOutArray, float *pOutArray)
>>        {
>>                sms_spectrumMag(sizeOutArray, pInArray, pOutArray);
>>        }
>>
>> %}
>>
>>
>> at this point,  have the new function my_spectrumMag that wraps
>> sms_spectrumMag() and provides arguments that can be typemapped using
>> numpy.i  Now, I don't want to have to call the function
>> my_spectrumMag() in python, I want to use the original name, I would
>> like to call the function as:
>>
>> sms_spectrumMag(numpyArray1, numpyArray2)
>>
>> But, trying to %rename my_spectrumMag to sms_spectrumMag does not
>> work, the original sms_spectrumMag gets called in python instead.
>> Trying to %ignore the original function first as follows removes the
>> sms_spectrumMag completely from the module and I am left with
>> my_spectrumMag:
>>
>> %ignore sms_spectrumMag;
>> %rename (sms_spectrumMag) my_spectrumMag;
>>
>>
>> Do you see my problem?
>>
>>
>> On Wed, Jan 7, 2009 at 8:58 AM, Matthieu Brucher
>> <matthieu.brucher at gmail.com> wrote:
>> > 2009/1/6 Rich E <reakinator at gmail.com>:
>> >> This helped immensely.  I feel like I am getting close to being able
>> >> to accomplish what I would like with SWIG: producing a python module
>> >> that can be very 'python-like', while co-existing with the c library
>> >> that is very 'c-like'.
>> >>
>> >> There is one question still remaining though, is it possible to make
>> >> the wrapped function have the same name still?  Using either
>> >> my_spectrumMag or spectrumMag means I have to create a number of
>> >> inconsistencies between the python module and the c library.  It is
>> >> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
>> >> wrapped one, with the same name.  But my attempts at doing this so far
>> >> have not compiled because of name conflictions.
>> >
>> > Ok course you can. The function is renamed only if you say so. Perhaps
>> > can you provide a small example of what doesn't work at the moment ?
>> >
>> >> Thanks for the help, I think you are doing great things with this
>> >> numpy interface/typemaps system.
>> >
>> > Matthieu
>> > --
>> > Information System Engineer, Ph.D.
>> > Website: http://matthieu-brucher.developpez.com/
>> > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> > _______________________________________________
>> > Numpy-discussion mailing list
>> > Numpy-discussion at scipy.org
>> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From bevan07 at gmail.com  Thu Jan  8 18:59:03 2009
From: bevan07 at gmail.com (Bevan Jenkins)
Date: Thu, 8 Jan 2009 23:59:03 +0000 (UTC)
Subject: [Numpy-discussion] Accumulate values that are below threshold
References: <loom.20090108T024310-329@post.gmane.org>
	<9457e7c80901072236ua4d0667j66fe0ed96f9fccdb@mail.gmail.com>
Message-ID: <loom.20090108T235456-864@post.gmane.org>

St?fan van der Walt <stefan <at> sun.ac.za> writes:

> 
> Hi Bevan
> 
> Since the number of output elements are unknown, I don't think you can
> implement this efficiently using arrays.  If your dataset isn't too
> large, a for-loop should do the trick.  Otherwise, you may have to run
> your code through Cython, which optimises for-loops around Python
> lists.
> 
> thresh = 1.0
> carry = 0
> output = []
> for idx, val in data:
>     carry += val
>     if (carry - thresh) >= -1e-15:
>         output.append((idx, carry))
>         carry = 0
> 
> The comparison line above, "(carry - thresh0 >= -1e-15", may look
> strange -- it basically just does "carry >= thresh".  For some reason
> I don't quite understand, when accumulating floats, it sometimes
> happens that "1.0 != 1.0", so I use 1e-15 as protection.
> 
> Regards
> St?fan
> 
> 2009/1/8 Bevan Jenkins <bevan07 <at> gmail.com>:

Stefan,

Thanks for your solution, it does exactly what i require.  At the moment I am 
just running through proof of concept stuff so the dataset is small.  If I 
start to run into issues with the real data, then that might be the push I 
need to look into Cython.

Bevan






From mexicalex at yahoo.com  Thu Jan  8 21:15:24 2009
From: mexicalex at yahoo.com (Alexandra Geddes)
Date: Thu, 8 Jan 2009 18:15:24 -0800 (PST)
Subject: [Numpy-discussion] checking array for NaN values.
Message-ID: <494682.97803.qm@web51511.mail.re2.yahoo.com>

Is there an easy way to check an array for NaN values?  'all(array)' regards NaN as true (because it's not 0).  I've tried all(array != nan) which didn't work either.  When i call locations which i know are NaN, it returns -1.#IND.  But if i try to call all(array != -1.#IND), python interprets the # as the start of a comment.

Any thoughts?

thanks,
alex.
Alexandra Geddes
UC Davis


      


From josef.pktd at gmail.com  Thu Jan  8 21:34:49 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 8 Jan 2009 21:34:49 -0500
Subject: [Numpy-discussion] checking array for NaN values.
In-Reply-To: <494682.97803.qm@web51511.mail.re2.yahoo.com>
References: <494682.97803.qm@web51511.mail.re2.yahoo.com>
Message-ID: <1cd32cbb0901081834w443f7e9va12231a49392769a@mail.gmail.com>

On Thu, Jan 8, 2009 at 9:15 PM, Alexandra Geddes <mexicalex at yahoo.com> wrote:
> Is there an easy way to check an array for NaN values?  'all(array)' regards NaN as true (because it's not 0).  I've tried all(array != nan) which didn't work either.  When i call locations which i know are NaN, it returns -1.#IND.  But if i try to call all(array != -1.#IND), python interprets the # as the start of a comment.
>

I use

>>> np.any(np.isnan(np.array([1,2,np.nan])))
True


Josef


From cournape at gmail.com  Fri Jan  9 00:21:48 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 9 Jan 2009 14:21:48 +0900
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <496663D5.4090005@stsci.edu>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
	<496663D5.4090005@stsci.edu>
Message-ID: <5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>

On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>
> I do not expect the trunk to always work.  I even expect it to have
> bugs.  However, I do not expect there to be test failures for known
> reasons that result in wasSuccessful() returning false.  This is a bad
> programming practice.  It creates work for people trying to figure out
> what is wrong when the answer is already know.

Well, I don't agree it is bad practice: it is not ideal, yes, but I
don't think using KnownFailure is much better. My rationale being that
known failures are almost never worked on because it does not bug
anyone anymore, and it is very easy to forget about them - AFAICS,
most numpy/scipy known failures have never been worked on after being
tagged as such. I don't think we have a good system for those cases,
be it known failure - or just failing.

I will tag them as known failure, since I am the only one against it, though :)

David


From cournape at gmail.com  Fri Jan  9 00:27:46 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 9 Jan 2009 14:27:46 +0900
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
	<496663D5.4090005@stsci.edu>
	<5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
Message-ID: <5b8d13220901082127n2fca120foe043f72b709cf9ec@mail.gmail.com>

On Fri, Jan 9, 2009 at 2:21 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>
>> I do not expect the trunk to always work.  I even expect it to have
>> bugs.  However, I do not expect there to be test failures for known
>> reasons that result in wasSuccessful() returning false.  This is a bad
>> programming practice.  It creates work for people trying to figure out
>> what is wrong when the answer is already know.
>
> Well, I don't agree it is bad practice: it is not ideal, yes, but I
> don't think using KnownFailure is much better. My rationale being that
> known failures are almost never worked on because it does not bug
> anyone anymore, and it is very easy to forget about them - AFAICS,
> most numpy/scipy known failures have never been worked on after being
> tagged as such. I don't think we have a good system for those cases,
> be it known failure - or just failing.
>
> I will tag them as known failure, since I am the only one against it, though :)

Done in r6308 - please tell me if something still does not work as expected,

David


From ezindy at gmail.com  Fri Jan  9 01:05:10 2009
From: ezindy at gmail.com (Egor Zindy)
Date: Fri, 9 Jan 2009 15:05:10 +0900
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <a3297bc90901081243g3c04e089i94417f7d3438d3a5@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
	<e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
	<a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>
	<c156de00901071713i31876ec3u84d39b2ac4f27ab3@mail.gmail.com>
	<a3297bc90901081243g3c04e089i94417f7d3438d3a5@mail.gmail.com>
Message-ID: <c156de00901082205q2149955jf5cba8ea987c9a3@mail.gmail.com>

Hello Rich,

I know what you mean. %inclusion of header files saves a lot of effort! So,
I had another play with the code (what holiday this turned out to be;) and
as long as the declarations in the .i file are made in the right order, it
should be possible to:
  * %include the header file
  * %ignore a sms_ function
  * %rename the function my_ to sms_
  * %inline the my_ function

I changed the .i file (attached) and re-ran the test, it works. Again, this
is on my XP/cygwin/mingw32 system, so it could need some tuning on a
different system!

In all this, not sure where is best to put the %exception statement, but
placement shouldn't be critical, because it concerns the my_ function rather
than the original (or renamed) sms_ function.

Regards,
Egor

On Fri, Jan 9, 2009 at 5:43 AM, Rich E <reakinator at gmail.com> wrote:

> I am using %includ "sms.h", which is what is wrapping all my
> functions.  Without doing this, I have to hand-wrap every function in
> the header file!
>
> Is there a way to exclude certain definitions from my c header file
> when using %include, so that I can hand wrap them instead?
>
> On Thu, Jan 8, 2009 at 2:13 AM, Egor Zindy <ezindy at gmail.com> wrote:
> > Hello Rich,
> >
> > This is very strange. I got to test my example again, as long as you
> don't
> > do a
> > %include "dftmagnitude.h"
> > somewhere in the dftmagnitude.i, it's perfectly possible to do a
> > %rename (sms_spectrumMag) my_spectrumMag;
> > (see dftmagnitude3.zip attached in my previous mail and this one).
> >
> > So things for you to check:
> >   * does the simple dftmagnitude3.zip compile on your system?
> >   * what version of SWIG are you using? (I used 1.3.36 provided with
> cygwin)
> >   * do you have a %include statement somewhere in your own .i file?
> >
> > Matthieu, if you read this, there's a complete example provided in
> > dftmagnitude3.zip.
> >   * Wrapped function sms_spectrumMag in dftmagnitude.c and .h
> >   * SWIG wrapper dftmagnitude.i uses %inline and %rename statements
> >   * Example uses a modified numpy.i (see the previous mails in the
> thread).
> >   * test example provided in test_dftmagnitude.py
> >
> > Haven't tested it under Linux, but under winxp/cygwin/mingw32, the
> following
> > works for me (in cygwin):
> >
> > $ python setup_dftmagnitude.py build -cmingw32 ; mv
> > build/lib.win32-2.5/_dftmagnitude.pyd .
> > $ python test_dftmagnitude.py
> >
> > Regards,
> > Egor
> >
> > --
> > My Python:
> > $ python -i
> > Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
> (Intel)]
> > on win32
> >
> > My SWIG:
> > $ swig -version
> >
> > SWIG Version 1.3.36
> >
> > Compiled with g++ [i686-pc-cygwin]
> > Please see http://www.swig.org for reporting bugs and further
> information
> >
> >
> >
> >
> > On Thu, Jan 8, 2009 at 1:43 AM, Rich E <reakinator at gmail.com> wrote:
> >>
> >> Here is my example, trying to wrap the function sms_spectrumMag that
> >> we have been dealing with:
> >>
> >> %apply (int DIM1, float* IN_ARRAY1) {(int sizeInArray, float*
> pInArray)};
> >> %apply (int DIM1, float* INPLACE_ARRAY1) {(int sizeOutArray, float*
> >> pOutArray)};
> >>
> >> %inline %{
> >>
> >>        void my_spectrumMag( int sizeInArray, float *pInArray, int
> >> sizeOutArray, float *pOutArray)
> >>        {
> >>                sms_spectrumMag(sizeOutArray, pInArray, pOutArray);
> >>        }
> >>
> >> %}
> >>
> >>
> >> at this point,  have the new function my_spectrumMag that wraps
> >> sms_spectrumMag() and provides arguments that can be typemapped using
> >> numpy.i  Now, I don't want to have to call the function
> >> my_spectrumMag() in python, I want to use the original name, I would
> >> like to call the function as:
> >>
> >> sms_spectrumMag(numpyArray1, numpyArray2)
> >>
> >> But, trying to %rename my_spectrumMag to sms_spectrumMag does not
> >> work, the original sms_spectrumMag gets called in python instead.
> >> Trying to %ignore the original function first as follows removes the
> >> sms_spectrumMag completely from the module and I am left with
> >> my_spectrumMag:
> >>
> >> %ignore sms_spectrumMag;
> >> %rename (sms_spectrumMag) my_spectrumMag;
> >>
> >>
> >> Do you see my problem?
> >>
> >>
> >> On Wed, Jan 7, 2009 at 8:58 AM, Matthieu Brucher
> >> <matthieu.brucher at gmail.com> wrote:
> >> > 2009/1/6 Rich E <reakinator at gmail.com>:
> >> >> This helped immensely.  I feel like I am getting close to being able
> >> >> to accomplish what I would like with SWIG: producing a python module
> >> >> that can be very 'python-like', while co-existing with the c library
> >> >> that is very 'c-like'.
> >> >>
> >> >> There is one question still remaining though, is it possible to make
> >> >> the wrapped function have the same name still?  Using either
> >> >> my_spectrumMag or spectrumMag means I have to create a number of
> >> >> inconsistencies between the python module and the c library.  It is
> >> >> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
> >> >> wrapped one, with the same name.  But my attempts at doing this so
> far
> >> >> have not compiled because of name conflictions.
> >> >
> >> > Ok course you can. The function is renamed only if you say so. Perhaps
> >> > can you provide a small example of what doesn't work at the moment ?
> >> >
> >> >> Thanks for the help, I think you are doing great things with this
> >> >> numpy interface/typemaps system.
> >> >
> >> > Matthieu
> >> > --
> >> > Information System Engineer, Ph.D.
> >> > Website: http://matthieu-brucher.developpez.com/
> >> > Blogs: http://matt.eifelle.com and
> http://blog.developpez.com/?blog=92
> >> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> >> > _______________________________________________
> >> > Numpy-discussion mailing list
> >> > Numpy-discussion at scipy.org
> >> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >> >
> >> _______________________________________________
> >> Numpy-discussion mailing list
> >> Numpy-discussion at scipy.org
> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090109/49214814/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dftmagnitudei.zip
Type: application/zip
Size: 781 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090109/49214814/attachment.zip>

From stefan at sun.ac.za  Fri Jan  9 02:19:09 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 09:19:09 +0200
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
	<496663D5.4090005@stsci.edu>
	<5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
Message-ID: <9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>

2009/1/9 David Cournapeau <cournape at gmail.com>:
> On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>
>> I do not expect the trunk to always work.  I even expect it to have
>> bugs.  However, I do not expect there to be test failures for known
>> reasons that result in wasSuccessful() returning false.  This is a bad
>> programming practice.  It creates work for people trying to figure out
>> what is wrong when the answer is already know.
>
> Well, I don't agree it is bad practice: it is not ideal, yes, but I
> don't think using KnownFailure is much better. My rationale being that
> known failures are almost never worked on because it does not bug
> anyone anymore, and it is very easy to forget about them - AFAICS,
> most numpy/scipy known failures have never been worked on after being
> tagged as such. I don't think we have a good system for those cases,
> be it known failure - or just failing.

I agree with you point of view, but I also have sympathy for
Cristopher's situation.  I thought a solution to both problems would
be if we could find an easy way of executing all skipped tests as if
they were never decorated.  Turns out nose already has this
functionality:

nosetests numpy --no-skip

I think we should urge developers to run the test suite this way, so
that we remain aware of failures, even if they are decorated.

Hope that helps,
St?fan


From animator333 at yahoo.com  Fri Jan  9 02:19:53 2009
From: animator333 at yahoo.com (Prashant Saxena)
Date: Fri, 9 Jan 2009 12:49:53 +0530 (IST)
Subject: [Numpy-discussion] replace array values efficiently
Message-ID: <100425.79053.qm@web94914.mail.in2.yahoo.com>

Hi,

I am new to numpy and getting my hands on slowly. 

How do you replace integers from strings in an integer array. (1D)

For example:

array = {1,1,1,2,3,3,4}

replace 1 with "apple"
replace 2 with "cheery"
replace 3 with "mango"
replace 4 with "banana"

I know the general solution, but I am looking for an efficient way, supported by numpy/scipy to do this kind of conversion as fast as possible.

Thanks

Prashant



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090109/0ac8212b/attachment.html>

From robert.kern at gmail.com  Fri Jan  9 02:29:17 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 01:29:17 -0600
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
	<496663D5.4090005@stsci.edu>
	<5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
	<9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>
Message-ID: <3d375d730901082329q448c8c86nab1ac1204fd14b1f@mail.gmail.com>

On Fri, Jan 9, 2009 at 01:19, St?fan van der Walt <stefan at sun.ac.za> wrote:

> I think we should urge developers to run the test suite this way, so
> that we remain aware of failures, even if they are decorated.

I don't think we should use unit tests as a bug tracker. We have Trac.
Each known-failing test (or group of related tests) should have a
ticket prioritized and scheduled appropriately.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From robert.kern at gmail.com  Fri Jan  9 02:33:22 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 01:33:22 -0600
Subject: [Numpy-discussion] replace array values efficiently
In-Reply-To: <100425.79053.qm@web94914.mail.in2.yahoo.com>
References: <100425.79053.qm@web94914.mail.in2.yahoo.com>
Message-ID: <3d375d730901082333p40c5cf01hd034ddb5efef4cac@mail.gmail.com>

On Fri, Jan 9, 2009 at 01:19, Prashant Saxena <animator333 at yahoo.com> wrote:
> Hi,
>
> I am new to numpy and getting my hands on slowly.
>
> How do you replace integers from strings in an integer array. (1D)
>
> For example:
>
> array = {1,1,1,2,3,3,4}
>
> replace 1 with "apple"
> replace 2 with "cheery"
> replace 3 with "mango"
> replace 4 with "banana"
>
> I know the general solution, but I am looking for an efficient way,
> supported by numpy/scipy to do this kind of conversion as fast as possible.

I'd actually use a dictionary for this:

In [1]: replacements = {1: 'apple', 2: 'cherry', 3: 'mango', 4: 'banana'}

In [2]: map(replacements.get, [1,1,1,2,3,3,4])
Out[2]: ['apple', 'apple', 'apple', 'cherry', 'mango', 'mango', 'banana']


But if you really want to use numpy for this:


In [3]: from numpy import *

In [4]: replacements_array = array([None, 'apple', 'cherry', 'mango',
'banana'], dtype=object)

In [5]: replacements_array[[1,1,1,2,3,3,4]]
Out[5]: array([apple, apple, apple, cherry, mango, mango, banana], dtype=object)


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Fri Jan  9 02:19:46 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 09 Jan 2009 16:19:46 +0900
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>	<49665407.5030505@stsci.edu>	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>	<49665E0B.3030009@stsci.edu>	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>	<496663D5.4090005@stsci.edu>	<5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
	<9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>
Message-ID: <4966FA92.3010405@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
> 2009/1/9 David Cournapeau <cournape at gmail.com>:
>   
>> On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley at stsci.edu> wrote:
>>     
>>> I do not expect the trunk to always work.  I even expect it to have
>>> bugs.  However, I do not expect there to be test failures for known
>>> reasons that result in wasSuccessful() returning false.  This is a bad
>>> programming practice.  It creates work for people trying to figure out
>>> what is wrong when the answer is already know.
>>>       
>> Well, I don't agree it is bad practice: it is not ideal, yes, but I
>> don't think using KnownFailure is much better. My rationale being that
>> known failures are almost never worked on because it does not bug
>> anyone anymore, and it is very easy to forget about them - AFAICS,
>> most numpy/scipy known failures have never been worked on after being
>> tagged as such. I don't think we have a good system for those cases,
>> be it known failure - or just failing.
>>     
>
> I agree with you point of view, but I also have sympathy for
> Cristopher's situation.

Yes, it is not a black and white situation - I first misunderstood
Christopher situation because of the given context of tracking numpy
changes. I can see why it is annoying - but it gives me important
information (like for example the fact that solaris does not have the
same formatting issues than linux and mac os X thanks to recent bug
reports).

As Robert said, BTS is supposedly a better system for this for this kind
of things - but at least for me, trac is so slow and painful to use that
I try to avoid it as much as possible.

David


From stefan at sun.ac.za  Fri Jan  9 02:37:32 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 09:37:32 +0200
Subject: [Numpy-discussion] Replacement for numpy.distutils.config.try_run
Message-ID: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>

Hi,

In distutils/config.py line 32 I see:

                      "Usage of try_run is deprecated: please do not \n" \
                      "use it anymore, and avoid configuration checks \n" \
                      "involving running executable on the target machine.\n" \

What is the recommended way of doing configuration checks now?

Regards
St?fan


From david at ar.media.kyoto-u.ac.jp  Fri Jan  9 02:32:37 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 09 Jan 2009 16:32:37 +0900
Subject: [Numpy-discussion] Replacement for
	numpy.distutils.config.try_run
In-Reply-To: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
Message-ID: <4966FD95.8000809@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
> Hi,
>
> In distutils/config.py line 32 I see:
>
>                       "Usage of try_run is deprecated: please do not \n" \
>                       "use it anymore, and avoid configuration checks \n" \
>                       "involving running executable on the target machine.\n" \
>
> What is the recommended way of doing configuration checks now?
>   

It only refers to try_run kind of checks (that is when you need to run
an executable on the machine). I added this when I had some problems
running it on windows 64 bits with python 2.6 - According to Martin
Loewis, try_run was not supposed to work everywhere, in particular in
cross-compilation contexts (which is relatively current for windows in
the context 32 vs 64 vs itanium).

It happened that in that particular case where it is used in numpy, I
found a way around it, but I would like to get rid of it completely at
some point.

David


From stefan at sun.ac.za  Fri Jan  9 02:51:03 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 09:51:03 +0200
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <3d375d730901082329q448c8c86nab1ac1204fd14b1f@mail.gmail.com>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
	<496663D5.4090005@stsci.edu>
	<5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
	<9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>
	<3d375d730901082329q448c8c86nab1ac1204fd14b1f@mail.gmail.com>
Message-ID: <9457e7c80901082351y6719b467p6fd156b4d9792c13@mail.gmail.com>

2009/1/9 Robert Kern <robert.kern at gmail.com>:
> On Fri, Jan 9, 2009 at 01:19, St?fan van der Walt <stefan at sun.ac.za> wrote:
>
>> I think we should urge developers to run the test suite this way, so
>> that we remain aware of failures, even if they are decorated.
>
> I don't think we should use unit tests as a bug tracker. We have Trac.
> Each known-failing test (or group of related tests) should have a
> ticket prioritized and scheduled appropriately.

Yup.  Point is, an implementation comes with tests, and sometimes
those tests break.  We then have two choices: remove the tests or mark
them as known failures.  Since they are meant to indicate regressions,
it would make little sense to remove them, hence these decorators.  My
suggestion was to use the --no-skip flag to make these decorators
invisible to the developers, lest failures be forgotten.

St?fan


From stefan at sun.ac.za  Fri Jan  9 03:04:05 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 10:04:05 +0200
Subject: [Numpy-discussion] change made to test_print.py
In-Reply-To: <4966FA92.3010405@ar.media.kyoto-u.ac.jp>
References: <49662BDF.7000806@stsci.edu>
	<5b8d13220901081115w19176d93y4cb4fea55aae1a53@mail.gmail.com>
	<49665407.5030505@stsci.edu>
	<5b8d13220901081148ge59c6a5x8bb929f312f6b9c3@mail.gmail.com>
	<49665E0B.3030009@stsci.edu>
	<5b8d13220901081226g1a0883ecg2d8e55d7b86bc17@mail.gmail.com>
	<496663D5.4090005@stsci.edu>
	<5b8d13220901082121h5806285x3422756eeec2a512@mail.gmail.com>
	<9457e7c80901082319v28122044h14f625e62cdf0bd9@mail.gmail.com>
	<4966FA92.3010405@ar.media.kyoto-u.ac.jp>
Message-ID: <9457e7c80901090004s5e0bb313web2ff117b8a7b8ec@mail.gmail.com>

2009/1/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> As Robert said, BTS is supposedly a better system for this for this kind
> of things - but at least for me, trac is so slow and painful to use that
> I try to avoid it as much as possible.

We are running Trac 10.2 from November 2006, so it is quite possible
that some of the speed issues have been addressed in the meantime.

Cheers
St?fan


From stefan at sun.ac.za  Fri Jan  9 03:05:19 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 10:05:19 +0200
Subject: [Numpy-discussion] Replacement for
	numpy.distutils.config.try_run
In-Reply-To: <4966FD95.8000809@ar.media.kyoto-u.ac.jp>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>
Message-ID: <9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>

2009/1/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> It happened that in that particular case where it is used in numpy, I
> found a way around it, but I would like to get rid of it completely at
> some point.

What do you suggest as workarounds?

St?fan


From david at ar.media.kyoto-u.ac.jp  Fri Jan  9 03:00:36 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 09 Jan 2009 17:00:36 +0900
Subject: [Numpy-discussion] Replacement
	for	numpy.distutils.config.try_run
In-Reply-To: <9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>
Message-ID: <49670424.7010108@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
> 2009/1/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>   
>> It happened that in that particular case where it is used in numpy, I
>> found a way around it, but I would like to get rid of it completely at
>> some point.
>>     
>
> What do you suggest as workarounds?
>
>   

What about not using tests which need to run on the target platform :)

It is not always easy, but in numpy cases, it is simple, at least
principle (numscons build does not run any test code, for example):
try_run is used to check whether some preprocessor defined are available
(see numpy/random and numpy/core), which is not necessary. The autobook
specifically mentions that code which need to run on target platforms
should be avoided, since it breaks cross-compilation; with python
distutils, it exactly breaks in those cases.

cheers,

David


From stefan at sun.ac.za  Fri Jan  9 03:31:25 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 10:31:25 +0200
Subject: [Numpy-discussion] Replacement for
	numpy.distutils.config.try_run
In-Reply-To: <49670424.7010108@ar.media.kyoto-u.ac.jp>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>
	<49670424.7010108@ar.media.kyoto-u.ac.jp>
Message-ID: <9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>

2009/1/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>> What do you suggest as workarounds?
>
> What about not using tests which need to run on the target platform :)

Let me simplify the question.  How do you detect the version of the
local Fortran compiler without executing the compiler?  Or is that OK,
and you'd simply like to avoid compiling and running code?

St?fan


From robert.kern at gmail.com  Fri Jan  9 03:37:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 02:37:02 -0600
Subject: [Numpy-discussion] Replacement for
	numpy.distutils.config.try_run
In-Reply-To: <9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>
	<49670424.7010108@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>
Message-ID: <3d375d730901090037n3b3ac24bp9781b7c4cd24af1c@mail.gmail.com>

On Fri, Jan 9, 2009 at 02:31, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2009/1/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>> What do you suggest as workarounds?
>>
>> What about not using tests which need to run on the target platform :)
>
> Let me simplify the question.  How do you detect the version of the
> local Fortran compiler without executing the compiler?

try_run() is not the right thing to call for such a purpose. Use
FCompiler.get_version().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From david at ar.media.kyoto-u.ac.jp  Fri Jan  9 03:21:47 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 09 Jan 2009 17:21:47 +0900
Subject: [Numpy-discussion] Replacement
	for	numpy.distutils.config.try_run
In-Reply-To: <9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>	<9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>	<49670424.7010108@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>
Message-ID: <4967091B.2080007@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
> 2009/1/9 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>   
>>> What do you suggest as workarounds?
>>>       
>> What about not using tests which need to run on the target platform :)
>>     
>
> Let me simplify the question.  How do you detect the version of the
> local Fortran compiler without executing the compiler?  Or is that OK,
> and you'd simply like to avoid compiling and running code?
>   

Ah, sorry, I used the autotools vocabulary you may not be familiar with:
when cross compiling, you have at least two platforms, build and
host/target (host/target is the same unless you build cross-compilers).
If I build foobar on linux for windows, linux is the build, windows the
host/target.

Anything which runs on the host/target platform cannot work in this
context; anything on the build of course can - which generally includes
compilers, etc... Basically, assuming a working compiler,
pre-processing, compiling, linking test code snippets is OK. Running is not.

cheers,

David


From stefan at sun.ac.za  Fri Jan  9 04:27:22 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 9 Jan 2009 11:27:22 +0200
Subject: [Numpy-discussion] Replacement for
	numpy.distutils.config.try_run
In-Reply-To: <3d375d730901090037n3b3ac24bp9781b7c4cd24af1c@mail.gmail.com>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>
	<49670424.7010108@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>
	<3d375d730901090037n3b3ac24bp9781b7c4cd24af1c@mail.gmail.com>
Message-ID: <9457e7c80901090127p31fde40at17cf2338e1339341@mail.gmail.com>

2009/1/9 Robert Kern <robert.kern at gmail.com>:
> try_run() is not the right thing to call for such a purpose. Use
> FCompiler.get_version().

That was just an example.  What I want to do is run something like
"pkg-config blah" and parse the output, but I get the idea from
David's post that that is OK.

Cheers
St?fan


From robert.kern at gmail.com  Fri Jan  9 04:30:01 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 03:30:01 -0600
Subject: [Numpy-discussion] Replacement for
	numpy.distutils.config.try_run
In-Reply-To: <9457e7c80901090127p31fde40at17cf2338e1339341@mail.gmail.com>
References: <9457e7c80901082337q7d1ba180ge52f64914a912036@mail.gmail.com>
	<4966FD95.8000809@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090005lc1911dve26e19b9793ea061@mail.gmail.com>
	<49670424.7010108@ar.media.kyoto-u.ac.jp>
	<9457e7c80901090031i3be213cfsc3182a678bddd2f9@mail.gmail.com>
	<3d375d730901090037n3b3ac24bp9781b7c4cd24af1c@mail.gmail.com>
	<9457e7c80901090127p31fde40at17cf2338e1339341@mail.gmail.com>
Message-ID: <3d375d730901090130j5f53c6f3v7343ac73cd694890@mail.gmail.com>

On Fri, Jan 9, 2009 at 03:27, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2009/1/9 Robert Kern <robert.kern at gmail.com>:
>> try_run() is not the right thing to call for such a purpose. Use
>> FCompiler.get_version().
>
> That was just an example.  What I want to do is run something like
> "pkg-config blah" and parse the output, but I get the idea from
> David's post that that is OK.

But that's not what try_run() does. Use exec_command() for general
purpose running of programs. try_run() compiles a program from source
and runs the result.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From nicolas.roux at st.com  Fri Jan  9 07:04:36 2009
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Fri, 9 Jan 2009 13:04:36 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <496501E1.2060400@molden.no>
Message-ID: <002d01c97252$716774a0$e7ad810a@gnb.st.com>

Hi !

Thanks a lot for your fast/detailed reply.
A very good point for Numpy ;-)

I spent all my time trying to prepare my testcase to better share with you,
that's why I didn't reply fast.

I understand the weakness of the missing JITcompiler in Python vs Matlab,
that's why I invistigated numpy vectorization/broadcast.
(hoping to find a cool way to write our code in fast Numpy)

I used the page http://www.scipy.org/PerformancePython to write my code
efficiently in Numpy.
While doing it I found one issue.

To have pretty code, I created p0 and p1 arrays of indexes.
In "test8" I wished to see the commented line working, which is not the
case. 
Having to use "ix_" is not pretty enough, and seems to not work with further
dimensions.
Why the comment line is not working ?

############################################
def test8():
    m = 1024
    n = 512
    
    Out = numpy.zeros((m,n))
    In = numpy.zeros((m,n))
  
    p0 = numpy.ogrid[0:m]
    p1 = numpy.ogrid[0:n]

    Out[0:m,0:n] = In[0:m,0:n]
    #Out[p0,p1] = In[p0,p1]   #This doesn't work
    Out[numpy.ix_(p0,p1)] = In[numpy.ix_(p0,p1)]
############################################

What is maybe not clear in the above code, is that I don't want to predefine
all possible ogrid/vector.
The number of possible ogrid/vector is big if in need to define all.
... And this vector definition become more paintful.

So Numpy vector style is fine if i can write something like:
   Out[p0,p1] = In[p0,p1]   #2 dimensions case
And 
   Out[p0,p1,1] = In[p0,p1,1] #3 dimensions case
But is not fine if i have to add ".ix_()" or to multiply the number of
vector definitions.

Below example with 3 dimensions instead of 2.
############################################
def test9():
    m = 1024
    n = 512
    
    Out = numpy.zeros((m,n,3))
    In = numpy.zeros((m,n,3))
  
    p0 = numpy.ogrid[0:m]
    p1 = numpy.ogrid[0:n]

    Out[0:m,0:n,2] = In[0:m,0:n,2]
    #Out[p0,p1,2] = In[p0,p1,2]
    Out[numpy.ix_(p0,p1,2)] = In[numpy.ix_(p0,p1,2)]
############################################

Tanks again for your support ;-)

Cheers,
Nicolas.



From ndbecker2 at gmail.com  Fri Jan  9 07:05:02 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 09 Jan 2009 07:05:02 -0500
Subject: [Numpy-discussion] memmap from fd?
Message-ID: <gk7ehe$unj$1@ger.gmane.org>

I'm working on interfacing to a custom FPGA board.  The kernel driver exposes the FPGA memory via mmap.

It might be nice to use numpy memmap to read/write data.  One issue is that I think I will need to create the memmap array from a fd, not a file name.  The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons.  Any chance to create a memmap array from a fd?



From robert.kern at gmail.com  Fri Jan  9 07:10:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 06:10:40 -0600
Subject: [Numpy-discussion] memmap from fd?
In-Reply-To: <gk7ehe$unj$1@ger.gmane.org>
References: <gk7ehe$unj$1@ger.gmane.org>
Message-ID: <3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>

On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2 at gmail.com> wrote:
> I'm working on interfacing to a custom FPGA board.  The kernel driver exposes the FPGA memory via mmap.
>
> It might be nice to use numpy memmap to read/write data.  One issue is that I think I will need to create the memmap array from a fd, not a file name.  The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons.  Any chance to create a memmap array from a fd?

Use os.fdopen(fd) to create a file object which can be passed to the
memmap constructor.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Fri Jan  9 08:26:00 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 09 Jan 2009 08:26 -0500
Subject: [Numpy-discussion] memmap from fd?
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
Message-ID: <gk7j99$h0j$1@ger.gmane.org>

Robert Kern wrote:

> On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2 at gmail.com> wrote:
>> I'm working on interfacing to a custom FPGA board.  The kernel driver
>> exposes the FPGA memory via mmap.
>>
>> It might be nice to use numpy memmap to read/write data.  One issue is
>> that I think I will need to create the memmap array from a fd, not a file
>> name.  The reason is I wrote the driver to only allow 1 exclusive open,
>> and I already have it open for other reasons.  Any chance to create a
>> memmap array from a fd?
> 
> Use os.fdopen(fd) to create a file object which can be passed to the
> memmap constructor.
> 

Thanks!  I'm assuming in this case I can ignore comments about flushing the data to disk?

If an assignment to a slice of an memmap array will call mmap, there should be no need for any flushing.



From sturla at molden.no  Fri Jan  9 08:50:29 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 9 Jan 2009 14:50:29 +0100 (CET)
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <002d01c97252$716774a0$e7ad810a@gnb.st.com>
References: <002d01c97252$716774a0$e7ad810a@gnb.st.com>
Message-ID: <696a2c95645b74e165e0d196a6f6582e.squirrel@webmail.uio.no>


> I understand the weakness of the missing JITcompiler in Python vs Matlab,
> that's why I invistigated numpy vectorization/broadcast.
> (hoping to find a cool way to write our code in fast Numpy)
>
> I used the page http://www.scipy.org/PerformancePython to write my code
> efficiently in Numpy.
> While doing it I found one issue.
>
> To have pretty code, I created p0 and p1 arrays of indexes.

I must admit I don't quite understand what you are trying to do, and what
your problem is.

If you just want to do

Out[:,:] = In[:,:]

there is no need for meshgrids (ogrid), for-loops, or whatever.

It is brain dead to use nested for-loops or ogrid for this purpose in
NumPy. It is equally foolish to use nested for loops or meshgrid for this
purpose in Matlab. If you do, I would seriously question your competence.

By the way, you can index ogrid with more than one dimension:

p = numpy.ogrid[:m, :n]
Out[p] = In[p]
















From ndbecker2 at gmail.com  Fri Jan  9 09:08:48 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 09 Jan 2009 09:08:48 -0500
Subject: [Numpy-discussion] memmap from fd?
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
Message-ID: <gk7lpr$q9j$1@ger.gmane.org>

Robert Kern wrote:

> On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2 at gmail.com> wrote:
>> I'm working on interfacing to a custom FPGA board.  The kernel driver
>> exposes the FPGA memory via mmap.
>>
>> It might be nice to use numpy memmap to read/write data.  One issue is
>> that I think I will need to create the memmap array from a fd, not a file
>> name.  The reason is I wrote the driver to only allow 1 exclusive open,
>> and I already have it open for other reasons.  Any chance to create a
>> memmap array from a fd?
> 
> Use os.fdopen(fd) to create a file object which can be passed to the
> memmap constructor.
> 

Looks like this is not going to work without some change to memmap.  The problem is, I need read/write access.  The only choice in memmap is 'w+'.

But this does:
        if (mode == 'w+') and shape is None:
            raise ValueError, "shape must be given"

        fid.seek(0,2)
 
My device has hijacked 'read' to mean something entirely different than you might expect.  The seek call invokes 'read'.

It looks like the purpose of this code is to find the size of the mappable area.  The best solution I think is just throw it away.  Consistent with mmap semantics, attempting access outside the mappable area should cause and error - but I don't think there is any reliable way to know the length of the mappable area apriori.

Any thoughts?



From nouiz at nouiz.org  Fri Jan  9 09:25:43 2009
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Fri, 9 Jan 2009 09:25:43 -0500
Subject: [Numpy-discussion] inplace matrix multiplication
Message-ID: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com>

Hi,

I would like to know how I can make a call to the blas function gemm in
numpy. I need a multiply and accumulate for matrix and I don't want to
allocate a new matrix each time I do it.

thanks for your time

Frederic Bastien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090109/413cedf2/attachment.html>

From nicolas.roux at st.com  Fri Jan  9 09:56:08 2009
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Fri, 9 Jan 2009 15:56:08 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <696a2c95645b74e165e0d196a6f6582e.squirrel@webmail.uio.no>
Message-ID: <012801c9726a$67af6ea0$e7ad810a@gnb.st.com>

Sorry my previous mail was probalby not clear.
This mail was following the tread we had before, so with some discussion
legacy.

I simplified the code to focus only on "what I" need, rather to bother you
with the full code.

I wrote below a code closer to what I need, where you will agree that
vectorization/broadcasting is needed to avoid nested loops.
As I wrote in the 1st mail (added at the end), what is important is to keep
the code not too ugly due to vectorization syntax.
(As written below I try to demonstrate that vectorization/broadcast code
could be as readable as twice nested loop )
The real code we have is even more complex, with processing the array
element using 5x5 neighbours, instead of 3x3.

######################################################
def test6():
    w = 3096
    h = 2048
    a = numpy.zeros((h,w))      #Normally loaded with real data
    b = numpy.zeros((h,w,3))    

    w0 = numpy.ogrid[0:w-2]
    w1 = numpy.ogrid[1:w-1]
    w2 = numpy.ogrid[2:w]

    h0 = numpy.ogrid[0:h-2]
    h1 = numpy.ogrid[1:h-1]
    h2 = numpy.ogrid[2:h]

    p00, p10, p20 = [h0,w0], [h1,w0],[h2,w0]
    p01, p11, p21 = [h0,w1], [h1,w1],[h2,w1]
    p02, p12, p22 = [h0,w2], [h1,w2],[h2,w2]         

    b[p11,1] =  a[p11] + 1.23*a[p22]  \
                     - numpy.min([a[p11]-a[p00],
                                  a[p11]-a[p01],
                                  a[p11]-a[p02],
                                  a[p11]-a[p10],
                                  a[p11]-a[p12],
                                  a[p11]-a[p20],
                                  a[p11]-a[p21],
                                  a[p11]-a[p22]])  \
                   + 0.123*numpy.max([a[p11]-a[p00],
                                      a[p11]-a[p01],
                                      a[p11]-a[p02],
                                      a[p11]-a[p10],
                                      a[p11]-a[p12],
                                      a[p11]-a[p20],
                                      a[p11]-a[p21],
                                      a[p11]-a[p22]])
######################################################

This code above is the one I wish to write but is not working.
I hope you better understand my issue context ;-)

Did I missed something  ?

Thanks for your help.

Cheers,
Nicolas.


> I understand the weakness of the missing JITcompiler in Python vs Matlab,
> that's why I invistigated numpy vectorization/broadcast.
> (hoping to find a cool way to write our code in fast Numpy)
>
> I used the page http://www.scipy.org/PerformancePython to write my code
> efficiently in Numpy.
> While doing it I found one issue.
>
> To have pretty code, I created p0 and p1 arrays of indexes.

I must admit I don't quite understand what you are trying to do, and what
your problem is.

If you just want to do

Out[:,:] = In[:,:]

there is no need for meshgrids (ogrid), for-loops, or whatever.

It is brain dead to use nested for-loops or ogrid for this purpose in
NumPy. It is equally foolish to use nested for loops or meshgrid for this
purpose in Matlab. If you do, I would seriously question your competence.

By the way, you can index ogrid with more than one dimension:

p = numpy.ogrid[:m, :n]
Out[p] = In[p]

============================================================================
===============
============================================================================
===============
============================================================================
===============

Hi !

Thanks a lot for your fast/detailed reply.
A very good point for Numpy ;-)

I spent all my time trying to prepare my testcase to better share with you,
that's why I didn't reply fast.

I understand the weakness of the missing JITcompiler in Python vs Matlab,
that's why I invistigated numpy vectorization/broadcast.
(hoping to find a cool way to write our code in fast Numpy)

I used the page http://www.scipy.org/PerformancePython to write my code
efficiently in Numpy.
While doing it I found one issue.

To have pretty code, I created p0 and p1 arrays of indexes.
In "test8" I wished to see the commented line working, which is not the
case. 
Having to use "ix_" is not pretty enough, and seems to not work with further
dimensions.
Why the comment line is not working ?

############################################
def test8():
    m = 1024
    n = 512
    
    Out = numpy.zeros((m,n))
    In = numpy.zeros((m,n))
  
    p0 = numpy.ogrid[0:m]
    p1 = numpy.ogrid[0:n]

    Out[0:m,0:n] = In[0:m,0:n]
    #Out[p0,p1] = In[p0,p1]   #This doesn't work
    Out[numpy.ix_(p0,p1)] = In[numpy.ix_(p0,p1)]
############################################

What is maybe not clear in the above code, is that I don't want to predefine
all possible ogrid/vector.
The number of possible ogrid/vector is big if in need to define all.
... And this vector definition become more paintful.

So Numpy vector style is fine if i can write something like:
   Out[p0,p1] = In[p0,p1]   #2 dimensions case
And 
   Out[p0,p1,1] = In[p0,p1,1] #3 dimensions case
But is not fine if i have to add ".ix_()" or to multiply the number of
vector definitions.

Below example with 3 dimensions instead of 2.
############################################
def test9():
    m = 1024
    n = 512
    
    Out = numpy.zeros((m,n,3))
    In = numpy.zeros((m,n,3))
  
    p0 = numpy.ogrid[0:m]
    p1 = numpy.ogrid[0:n]

    Out[0:m,0:n,2] = In[0:m,0:n,2]
    #Out[p0,p1,2] = In[p0,p1,2]
    Out[numpy.ix_(p0,p1,2)] = In[numpy.ix_(p0,p1,2)]
############################################

Tanks again for your support ;-)

Cheers,
Nicolas.


============================================================================
===============
============================================================================
===============
============================================================================
===============
Hi,

I need help ;-)
I have here a testcase which works much faster in Matlab than Numpy.
 
The following code takes less than 0.9sec in Matlab, but 21sec in Python.
Numpy is 24 times slower than Matlab !
The big trouble I have is a large team of people within my company is ready
to replace Matlab by Numpy/Scipy/Matplotlib,
but I have to demonstrate that this kind of Python Code is executed with the
same performance than Matlab, without writing C extension.
This is becoming a critical point for us.

This is a testcase that people would like to see working without any code
restructuring.
The reasons are:
- this way of writing is fairly natural.
- the original code which showed me the matlab/Numpy performance differences
is much more complex,
and can't benefit from broadcasting or other numpy tips (I can later give
this code)

...So I really need to use the code below, without restructuring.

Numpy/Python code:
#####################################################################
import numpy
import time

print "Start test \n" 

dim = 3000

a = numpy.zeros((dim,dim,3))

start = time.clock()

for i in range(dim):
    for j in range(dim):
        a[i,j,0] = a[i,j,1]
        a[i,j,2] = a[i,j,0]
        a[i,j,1] = a[i,j,2]

end = time.clock() - start        

print "Test done,   %f sec" % end
#####################################################################

Matlab code:
#####################################################################
'Start test'
dim = 3000;
tic;
a =zeros(dim,dim,3);
for i = 1:dim
    for j = 1:dim 
        a(i,j,1) = a(i,j,2);
        a(i,j,2) = a(i,j,1);
        a(i,j,3) = a(i,j,3);
    end
end
toc
'Test done'
#####################################################################

Any idea on it ?
Did I missed something ?

Thanks a lot, in advance for your help.


Cheers,
Nicolas.







_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From sturla at molden.no  Fri Jan  9 10:46:43 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 9 Jan 2009 16:46:43 +0100 (CET)
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <012801c9726a$67af6ea0$e7ad810a@gnb.st.com>
References: <012801c9726a$67af6ea0$e7ad810a@gnb.st.com>
Message-ID: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>


> I simplified the code to focus only on "what I" need, rather to bother you
> with the full code.

def test():
    w = 3096
    h = 2048
    a = numpy.zeros((h,w), order='F')      #Normally loaded with real data
    b = numpy.zeros((h,w,3), order='F')

    w0 = slice(0,w-2)
    w1 = slice(1,w-1)
    w2 = slice(2,w)
    h0 = slice(0,h-2)
    h1 = slice(1,h-1)
    h2 = slice(2,h)

    p00, p10, p20 = [h0,w0], [h1,w0], [h2,w0]
    p01, p11, p21 = [h0,w1], [h1,w1], [h2,w1]
    p02, p12, p22 = [h0,w2], [h1,w2], [h2,w2]

    b[p11 + [1]] =  a[p11] + 1.23*a[p22]  \
                     - numpy.min([a[p11]-a[p00],
                                  a[p11]-a[p01],
                                  a[p11]-a[p02],
                                  a[p11]-a[p10],
                                  a[p11]-a[p12],
                                  a[p11]-a[p20],
                                  a[p11]-a[p21],
                                  a[p11]-a[p22]])  \
                   + 0.123*numpy.max([a[p11]-a[p00],
                                      a[p11]-a[p01],
                                      a[p11]-a[p02],
                                      a[p11]-a[p10],
                                      a[p11]-a[p12],
                                      a[p11]-a[p20],
                                      a[p11]-a[p21],
                                      a[p11]-a[p22]])


Does this work for you?





From ndbecker2 at gmail.com  Fri Jan  9 11:59:58 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 09 Jan 2009 11:59:58 -0500
Subject: [Numpy-discussion] memmap problem
Message-ID: <gk7vqf$1uq$1@ger.gmane.org>

I modified memmap.py to avoid the issues with needed to read.  It is working, but I am seeing these:

 m
Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([255, 255, 255], dtype=uint8)> ignored
Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0], dtype=uint8)> ignored
Out[22]: eos_memmap([  0,   0,   0, ..., 255, 255, 255], dtype=uint8)

print m[0:4]
Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0, 0], dtype=uint8)> ignored
[0 0 0 0]

What's that about?



From nicolas.roux at st.com  Fri Jan  9 12:32:50 2009
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Fri, 9 Jan 2009 18:32:50 +0100
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
Message-ID: <015501c97280$4c036100$e7ad810a@gnb.st.com>

Thanks !

-1- The code style is good and the performance vs matlab is good.
With 400x400:
 Matlab = 1.56 sec  (with nested "for" loop, so no optimization)
 Numpy  = 0.99 sec  (with broadcasting)


-2- Now with the code below I have strange result.
With w=h=400:
   - Using "slice"        =>  0.99 sec
   - Using "numpy.ogrid"  =>  0.01 sec

With w=400 and h=300:
   - Using "slice",       => 0.719sec
   - Using "numpy.ogrid", => broadcast ERROR !
 
The last broadcast error is:
 "ValueError: shape mismatch: objects cannot be broadcast to a single shape"

#######################################################
def test():
    w = 400
 
    if 1:  #---Case with different w and h
        h = 300
    else:  #---Case with same w and h
        h = 400

    a = numpy.zeros((h,w))      #Normally loaded with real data
    b = numpy.zeros((h,w,3))

    if 1:   #---Case with SLICE
        w0 = slice(0,w-2)
        w1 = slice(1,w-1)
        w2 = slice(2,w)
        h0 = slice(0,h-2)
        h1 = slice(1,h-1)
        h2 = slice(2,h)
    else:   #---Case with OGRID
        w0 = numpy.ogrid[0:w-2]
        w1 = numpy.ogrid[1:w-1]
        w2 = numpy.ogrid[2:w]
        h0 = numpy.ogrid[0:h-2]
        h1 = numpy.ogrid[1:h-1]
        h2 = numpy.ogrid[2:h]

    p00, p01, p02 = [h0,w0], [h0,w1],[h0,w2]
    p10, p11, p12 = [h1,w0], [h1,w1],[h1,w2]
    p20, p21, p22 = [h2,w0], [h2,w1],[h2,w2]         

    b[p11+[1]] =  a[p11] - numpy.min([a[p11]-a[p00],
                                  a[p11]-a[p01],
                                  a[p11]-a[p02],
                                  a[p11]-a[p10],
                                  a[p11]-a[p12],
                                  a[p11]-a[p20],
                                  a[p11]-a[p21],
                                  a[p11]-a[p22]])  \
                   + 0.123*numpy.max([a[p11]-a[p00],
                                      a[p11]-a[p01],
                                      a[p11]-a[p02],
                                      a[p11]-a[p10],
                                      a[p11]-a[p12],
                                      a[p11]-a[p20],
                                      a[p11]-a[p21],
                                      a[p11]-a[p22]])
#######################################################

It seems "ogrid" got better performance, but broadcasting is not working any
more.
Do you think it's normal that broadcast is not possible using ogrid and
different w & h ?
Did I missed any row/colomn missmatch ???

Thanks.

Cheers,
Nicolas



-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Sturla Molden
Sent: Friday, January 09, 2009 4:47 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy performance vs Matlab.


> I simplified the code to focus only on "what I" need, rather to bother you
> with the full code.

def test():
    w = 3096
    h = 2048
    a = numpy.zeros((h,w), order='F')      #Normally loaded with real data
    b = numpy.zeros((h,w,3), order='F')

    w0 = slice(0,w-2)
    w1 = slice(1,w-1)
    w2 = slice(2,w)
    h0 = slice(0,h-2)
    h1 = slice(1,h-1)
    h2 = slice(2,h)

    p00, p10, p20 = [h0,w0], [h1,w0], [h2,w0]
    p01, p11, p21 = [h0,w1], [h1,w1], [h2,w1]
    p02, p12, p22 = [h0,w2], [h1,w2], [h2,w2]

    b[p11 + [1]] =  a[p11] + 1.23*a[p22]  \
                     - numpy.min([a[p11]-a[p00],
                                  a[p11]-a[p01],
                                  a[p11]-a[p02],
                                  a[p11]-a[p10],
                                  a[p11]-a[p12],
                                  a[p11]-a[p20],
                                  a[p11]-a[p21],
                                  a[p11]-a[p22]])  \
                   + 0.123*numpy.max([a[p11]-a[p00],
                                      a[p11]-a[p01],
                                      a[p11]-a[p02],
                                      a[p11]-a[p10],
                                      a[p11]-a[p12],
                                      a[p11]-a[p20],
                                      a[p11]-a[p21],
                                      a[p11]-a[p22]])


Does this work for you?



_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pjssilva at ime.usp.br  Fri Jan  9 13:45:19 2009
From: pjssilva at ime.usp.br (Paulo J. S. Silva)
Date: Fri, 09 Jan 2009 15:45:19 -0300
Subject: [Numpy-discussion] 2-D function and meshgrid
Message-ID: <1231526719.28818.19.camel@trinity>

Hello,

I have a function that receives a array of shape (2,) and returns a
number (a function from R^2 -> R). It basically looks like this:

    def weirdDistance2(x):
       return dot(dot(weirdMatrix, x), x)

(weirdMatrix is a "global" (2,2) array)

I want to see its level sets in the box [0, 1] x [0, 1], hence I have to
create a meshgrid and then compute it at each point of the mesh:

    x = linspace(0, 1, 200)
    y = x.copy()
    X, Y = meshgrid(x, y)

My problem is how to actually compute the function at each point of the
mesh. I have come out with two solutions. One very short and clear, but
slow, and another longer and more convoluted (it has a loop, I hate
loops in numpy code), but faster. Does anyone know a "no explicit-loops"
and fast solution?

Solution1:

    def myDistance(a, b):
        return weirdDistance(np.array((a, b)))
    vecDistance = np.vectorize(myDistance)
    return vecDistance(X, Y)

Solution 2:

    nPoints = X.size
    result = np.zeros(nPoints)
    points = np.array( [X.ravel(), Y.ravel()] ).T
    for i in xrange(nPoints):
        result[i] = weirdDistance(points[i])
    result = result.reshape(X.shape)

Of course, the first one is slow because the myDistance function creates
an array at each call. The second one, even with a loop, avoids the
array creations.

Best,

Paulo



From Chris.Barker at noaa.gov  Fri Jan  9 14:13:36 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 09 Jan 2009 11:13:36 -0800
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <015501c97280$4c036100$e7ad810a@gnb.st.com>
References: <015501c97280$4c036100$e7ad810a@gnb.st.com>
Message-ID: <4967A1E0.2040208@noaa.gov>

Nicolas ROUX wrote:
> -2- Now with the code below I have strange result.
> With w=h=400:

> With w=400 and h=300:
>    - Using "numpy.ogrid", => broadcast ERROR !
>  
> The last broadcast error is:
>  "ValueError: shape mismatch: objects cannot be broadcast to a single shape"

This is probably a broadcasting error, which means your result with w=h 
is probably wrong.

My advice:

Always test with non-square arrays, to make sure you are broadcasting as 
you expect.

Don't test with "zeros" or "ones" for your test data -- so you can look 
at the results and see if you are getting what you expect. I often use 
something like:

a = numpy.arange(w*h).reshape((h,w))

I also test with very small dimensions so that I can easily print the 
results of each step as I develop the code

With this approach you will probably figure out what's going on.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From sturla at molden.no  Fri Jan  9 14:44:03 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 9 Jan 2009 20:44:03 +0100 (CET)
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <015501c97280$4c036100$e7ad810a@gnb.st.com>
References: <015501c97280$4c036100$e7ad810a@gnb.st.com>
Message-ID: <d7ff5ff0cb1280179814278de08e4ff4.squirrel@webmail.uio.no>


> -2- Now with the code below I have strange result.
> With w=h=400:
>    - Using "slice"        =>  0.99 sec
>    - Using "numpy.ogrid"  =>  0.01 sec

It is not equivalent. The ogrid version only uses diagonal elements, and
does less work.


> It seems "ogrid" got better performance, but broadcasting is not working
> any
> more.

Broadcasting is working, but not the way you think. Ogrid is not a faster
alternative to slicing.

You have the same in Matlab. You can index with a slice, an array of
indices, or an array of booleans. If you are going to use the second
alternative, the shape of the index arrays -- in each dimension -- must
equal that of the output. You cannot use a "meshgrid" with different
shaped arrays of x, y and z indices. NumPy is no different from Matlab
here.






From charlesr.harris at gmail.com  Fri Jan  9 15:37:50 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 9 Jan 2009 13:37:50 -0700
Subject: [Numpy-discussion] 2-D function and meshgrid
In-Reply-To: <1231526719.28818.19.camel@trinity>
References: <1231526719.28818.19.camel@trinity>
Message-ID: <e06186140901091237q1d680d3sb5fd06b0924def52@mail.gmail.com>

On Fri, Jan 9, 2009 at 11:45 AM, Paulo J. S. Silva <pjssilva at ime.usp.br>wrote:

> Hello,
>
> I have a function that receives a array of shape (2,) and returns a
> number (a function from R^2 -> R). It basically looks like this:
>
>    def weirdDistance2(x):
>       return dot(dot(weirdMatrix, x), x)
>
> (weirdMatrix is a "global" (2,2) array)
>
> I want to see its level sets in the box [0, 1] x [0, 1], hence I have to
> create a meshgrid and then compute it at each point of the mesh:
>
>    x = linspace(0, 1, 200)
>    y = x.copy()
>    X, Y = meshgrid(x, y)
>
> My problem is how to actually compute the function at each point of the
> mesh. I have come out with two solutions. One very short and clear, but
> slow, and another longer and more convoluted (it has a loop, I hate
> loops in numpy code), but faster. Does anyone know a "no explicit-loops"
> and fast solution?
>
> Solution1:
>
>    def myDistance(a, b):
>        return weirdDistance(np.array((a, b)))
>    vecDistance = np.vectorize(myDistance)
>    return vecDistance(X, Y)
>
> Solution 2:
>
>    nPoints = X.size
>    result = np.zeros(nPoints)
>    points = np.array( [X.ravel(), Y.ravel()] ).T
>    for i in xrange(nPoints):
>        result[i] = weirdDistance(points[i])
>    result = result.reshape(X.shape)
>
> Of course, the first one is slow because the myDistance function creates
> an array at each call. The second one, even with a loop, avoids the
> array creations.
>

Try

IDLE 1.2.4
>>> import numpy as np
>>> pts = np.random.rand(5,2)
>>> mat = np.random.rand(2,2)
>>> res = (np.dot(pts,mat)*pts).sum(axis=1)
>>> res

array([ 0.63018561,  0.30829864,  0.23173343,  1.79972127,  0.69498856])
>>> for row in pts : np.dot(row,np.dot(mat,row))

0.63018560596590589
0.30829864146737423
0.23173343333294744
1.7997212735553192
0.69498855520540959

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090109/2c4925d9/attachment.html>

From robert.kern at gmail.com  Fri Jan  9 15:53:48 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 14:53:48 -0600
Subject: [Numpy-discussion] memmap problem
In-Reply-To: <gk7vqf$1uq$1@ger.gmane.org>
References: <gk7vqf$1uq$1@ger.gmane.org>
Message-ID: <3d375d730901091253n597a294au13d9c32ef76463e5@mail.gmail.com>

On Fri, Jan 9, 2009 at 10:59, Neal Becker <ndbecker2 at gmail.com> wrote:
> I modified memmap.py to avoid the issues with needed to read.  It is working, but I am seeing these:
>
>  m
> Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([255, 255, 255], dtype=uint8)> ignored
> Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0], dtype=uint8)> ignored
> Out[22]: eos_memmap([  0,   0,   0, ..., 255, 255, 255], dtype=uint8)
>
> print m[0:4]
> Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0, 0], dtype=uint8)> ignored
> [0 0 0 0]
>
> What's that about?

Can you show us your modifications? You may need to also modify
__array_finalize__(), _close(), and __del__().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan  9 16:10:04 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 15:10:04 -0600
Subject: [Numpy-discussion] memmap from fd?
In-Reply-To: <gk7lpr$q9j$1@ger.gmane.org>
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
	<gk7lpr$q9j$1@ger.gmane.org>
Message-ID: <3d375d730901091310v27033d5co1fbdc71098b9e672@mail.gmail.com>

On Fri, Jan 9, 2009 at 08:08, Neal Becker <ndbecker2 at gmail.com> wrote:
> Robert Kern wrote:
>
>> On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> I'm working on interfacing to a custom FPGA board.  The kernel driver
>>> exposes the FPGA memory via mmap.
>>>
>>> It might be nice to use numpy memmap to read/write data.  One issue is
>>> that I think I will need to create the memmap array from a fd, not a file
>>> name.  The reason is I wrote the driver to only allow 1 exclusive open,
>>> and I already have it open for other reasons.  Any chance to create a
>>> memmap array from a fd?
>>
>> Use os.fdopen(fd) to create a file object which can be passed to the
>> memmap constructor.
>>
>
> Looks like this is not going to work without some change to memmap.  The problem is, I need read/write access.  The only choice in memmap is 'w+'.

'r+' is for reading and writing.

> But this does:
>        if (mode == 'w+') and shape is None:
>            raise ValueError, "shape must be given"
>
>        fid.seek(0,2)
>
> My device has hijacked 'read' to mean something entirely different than you might expect.  The seek call invokes 'read'.
>
> It looks like the purpose of this code is to find the size of the mappable area.  The best solution I think is just throw it away.

We can't. We need it.

> Consistent with mmap semantics, attempting access outside the mappable area should cause and error - but I don't think there is any reliable way to know the length of the mappable area apriori.

For regular files, that seems to me to be fairly reliable. Why isn't it?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan  9 16:20:59 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 15:20:59 -0600
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <015501c97280$4c036100$e7ad810a@gnb.st.com>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
Message-ID: <3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>

On Fri, Jan 9, 2009 at 11:32, Nicolas ROUX <nicolas.roux at st.com> wrote:
> Thanks !
>
> -1- The code style is good and the performance vs matlab is good.
> With 400x400:
>  Matlab = 1.56 sec  (with nested "for" loop, so no optimization)
>  Numpy  = 0.99 sec  (with broadcasting)
>
>
> -2- Now with the code below I have strange result.
> With w=h=400:
>   - Using "slice"        =>  0.99 sec
>   - Using "numpy.ogrid"  =>  0.01 sec
>
> With w=400 and h=300:
>   - Using "slice",       => 0.719sec
>   - Using "numpy.ogrid", => broadcast ERROR !
>
> The last broadcast error is:
>  "ValueError: shape mismatch: objects cannot be broadcast to a single shape"
>
> #######################################################
> def test():
>    w = 400
>
>    if 1:  #---Case with different w and h
>        h = 300
>    else:  #---Case with same w and h
>        h = 400
>
>    a = numpy.zeros((h,w))      #Normally loaded with real data
>    b = numpy.zeros((h,w,3))
>
>    if 1:   #---Case with SLICE
>        w0 = slice(0,w-2)
>        w1 = slice(1,w-1)
>        w2 = slice(2,w)
>        h0 = slice(0,h-2)
>        h1 = slice(1,h-1)
>        h2 = slice(2,h)
>    else:   #---Case with OGRID
>        w0 = numpy.ogrid[0:w-2]
>        w1 = numpy.ogrid[1:w-1]
>        w2 = numpy.ogrid[2:w]
>        h0 = numpy.ogrid[0:h-2]
>        h1 = numpy.ogrid[1:h-1]
>        h2 = numpy.ogrid[2:h]
>
>    p00, p01, p02 = [h0,w0], [h0,w1],[h0,w2]
>    p10, p11, p12 = [h1,w0], [h1,w1],[h1,w2]
>    p20, p21, p22 = [h2,w0], [h2,w1],[h2,w2]
>
>    b[p11+[1]] =  a[p11] - numpy.min([a[p11]-a[p00],
>                                  a[p11]-a[p01],
>                                  a[p11]-a[p02],
>                                  a[p11]-a[p10],
>                                  a[p11]-a[p12],
>                                  a[p11]-a[p20],
>                                  a[p11]-a[p21],
>                                  a[p11]-a[p22]])  \
>                   + 0.123*numpy.max([a[p11]-a[p00],
>                                      a[p11]-a[p01],
>                                      a[p11]-a[p02],
>                                      a[p11]-a[p10],
>                                      a[p11]-a[p12],
>                                      a[p11]-a[p20],
>                                      a[p11]-a[p21],
>                                      a[p11]-a[p22]])
> #######################################################
>
> It seems "ogrid" got better performance, but broadcasting is not working any
> more.
> Do you think it's normal that broadcast is not possible using ogrid and
> different w & h ?
> Did I missed any row/colomn missmatch ???

There are several things wrong. Please read this document for
information about how indexing works in numpy.

  http://docs.scipy.org/doc/numpy/user/basics.indexing.html

But basically, you want slices. Using ogrid correctly will be slower.
FWIW, ogrid with only one argument is fairly pointless. ogrid is
intended to be used with multiple dimensions. If you just need one
argument, use arange() or linspace(). With multiple arguments, ogrid
will align the arrays such that they can be broadcasted as you expect.
Lets take a look at some examples:

In [1]: from numpy import *

In [2]: ogrid[0:5]
Out[2]: array([0, 1, 2, 3, 4])

In [3]: ogrid[0:6]
Out[3]: array([0, 1, 2, 3, 4, 5])


Two 1D arrays. Now, if you follow the discussion in the indexing
document, you know that if I were to use these as index arrays, one
for each axis, the indexing mechanism will try to iterate over them in
parallel. Since they have incompatible shapes, this will fail.

Instead, if you put both arguments into ogrid:

In [4]: ogrid[0:5, 0:6]
Out[4]:
[array([[0],
       [1],
       [2],
       [3],
       [4]]),
 array([[0, 1, 2, 3, 4, 5]])]


We get the kind of arrays you need. These shapes are compatible,
through broadcasting, and together form the indices to select out the
part of the matrix you are interested in.

However, just using the slices on the matrix instead of passing the
slices through ogrid is faster.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan  9 16:31:36 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 15:31:36 -0600
Subject: [Numpy-discussion] inplace matrix multiplication
In-Reply-To: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com>
References: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com>
Message-ID: <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com>

On Fri, Jan 9, 2009 at 08:25, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> Hi,
>
> I would like to know how I can make a call to the blas function gemm in
> numpy. I need a multiply and accumulate for matrix and I don't want to
> allocate a new matrix each time I do it.

You can't in numpy. With scipy.linalg.fblas.dgemm() and the right
arguments, you can.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From Chris.Barker at noaa.gov  Fri Jan  9 16:40:25 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 09 Jan 2009 13:40:25 -0800
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
	<3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
Message-ID: <4967C449.9060508@noaa.gov>

Robert Kern wrote:
> Instead, if you put both arguments into ogrid:
> 
> In [4]: ogrid[0:5, 0:6]
> Out[4]:
> [array([[0],
>        [1],
>        [2],
>        [3],
>        [4]]),
>  array([[0, 1, 2, 3, 4, 5]])]
> 
> We get the kind of arrays you need. These shapes are compatible,
> through broadcasting, and together form the indices to select out the
> part of the matrix you are interested in.
> 
> However, just using the slices on the matrix instead of passing the
> slices through ogrid is faster.

So what is ogrid useful for?

Just curious...

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Fri Jan  9 16:47:04 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 15:47:04 -0600
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4967C449.9060508@noaa.gov>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
	<3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
	<4967C449.9060508@noaa.gov>
Message-ID: <3d375d730901091347s77f0a9f7l4898c8d9bd77b90b@mail.gmail.com>

On Fri, Jan 9, 2009 at 15:40, Christopher Barker <Chris.Barker at noaa.gov> wrote:

> So what is ogrid useful for?
>
> Just curious...

Floating point grids.

x, y = ogrid[0:1:101j, 0:1:101j]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From sturla at molden.no  Fri Jan  9 16:49:27 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 9 Jan 2009 22:49:27 +0100 (CET)
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4967C449.9060508@noaa.gov>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
	<3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
	<4967C449.9060508@noaa.gov>
Message-ID: <b694391a022f8a0d37ea004e5fcc3a5a.squirrel@webmail.uio.no>


>> However, just using the slices on the matrix instead of passing the
>> slices through ogrid is faster.
>
> So what is ogrid useful for?

For the same problems where you would use meshgrid in Matlab. That is
certain graphics problem for example; e.g. evaluating a surface z = f(x,y)
over a grid of x,y values.








From Chris.Barker at noaa.gov  Fri Jan  9 17:04:37 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 09 Jan 2009 14:04:37 -0800
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <b694391a022f8a0d37ea004e5fcc3a5a.squirrel@webmail.uio.no>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
	<3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
	<4967C449.9060508@noaa.gov>
	<b694391a022f8a0d37ea004e5fcc3a5a.squirrel@webmail.uio.no>
Message-ID: <4967C9F5.8070704@noaa.gov>

Sturla Molden wrote:
> For the same problems where you would use meshgrid in Matlab.

well, I used to use meshgrid a lot because MATLAB could not do 
broadcasting. Which is probably why the OP has been trying to use it.

A note for the docs:

The docs refer to ogrid and nd_grid, and as far as I can tell, they are 
the same thing, but it's confusing. actuall, as I look more, ogrid is a 
nd_grid with the sparse parameter set to True -- anyway, still confusing!

Also, what is the "o" for -- the mnemonic might be helpful

thanks,

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From reakinator at gmail.com  Fri Jan  9 17:11:04 2009
From: reakinator at gmail.com (Rich E)
Date: Fri, 9 Jan 2009 23:11:04 +0100
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <c156de00901082205q2149955jf5cba8ea987c9a3@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<c156de00901050142p3f45c30byc2c905a976331060@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
	<e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
	<a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>
	<c156de00901071713i31876ec3u84d39b2ac4f27ab3@mail.gmail.com>
	<a3297bc90901081243g3c04e089i94417f7d3438d3a5@mail.gmail.com>
	<c156de00901082205q2149955jf5cba8ea987c9a3@mail.gmail.com>
Message-ID: <a3297bc90901091411w670158aas404cf14fb8d9f577@mail.gmail.com>

Well I see it works, however with one change: the %apply typemaps need
to be done before %include'ing the header file, or else nothing in
that header file will automatically get typemapped (only the functions
that are written using %inline will be typemapped, which in the case
of the exampe you wrote, is all there is.  This isn't the case in the
library I am writing, there are about 20-30 others that don't need to
be written inline).

One comment though, upon looking at the wrapped c file: there are two
definitions for sms_spectrumMag(), one that expects pointers to c
arrays as arguments, and then one later that expects numpy arrays as
arguments.  Upon testing the function in python, it seems there is no
conflict; supplying just a numpy array works perfectly fine.  I dont'
understand how python is interpreting this so I cannot foresee any
problems, I'm just bringing it up in case others can.

So now I can wrap the functions in a way where they are very useful in
python without messing up the c code.  Great, thank you Egor for the
ongoing help!  Sorry it took your vacation, but you helped me spend
mine the way I wanted to (successful programming, hehe).

cheers,
Rich

On Fri, Jan 9, 2009 at 7:05 AM, Egor Zindy <ezindy at gmail.com> wrote:
> Hello Rich,
>
> I know what you mean. %inclusion of header files saves a lot of effort! So,
> I had another play with the code (what holiday this turned out to be;) and
> as long as the declarations in the .i file are made in the right order, it
> should be possible to:
>   * %include the header file
>   * %ignore a sms_ function
>   * %rename the function my_ to sms_
>   * %inline the my_ function
>
> I changed the .i file (attached) and re-ran the test, it works. Again, this
> is on my XP/cygwin/mingw32 system, so it could need some tuning on a
> different system!
>
> In all this, not sure where is best to put the %exception statement, but
> placement shouldn't be critical, because it concerns the my_ function rather
> than the original (or renamed) sms_ function.
>
> Regards,
> Egor
>
> On Fri, Jan 9, 2009 at 5:43 AM, Rich E <reakinator at gmail.com> wrote:
>>
>> I am using %includ "sms.h", which is what is wrapping all my
>> functions.  Without doing this, I have to hand-wrap every function in
>> the header file!
>>
>> Is there a way to exclude certain definitions from my c header file
>> when using %include, so that I can hand wrap them instead?
>>
>> On Thu, Jan 8, 2009 at 2:13 AM, Egor Zindy <ezindy at gmail.com> wrote:
>> > Hello Rich,
>> >
>> > This is very strange. I got to test my example again, as long as you
>> > don't
>> > do a
>> > %include "dftmagnitude.h"
>> > somewhere in the dftmagnitude.i, it's perfectly possible to do a
>> > %rename (sms_spectrumMag) my_spectrumMag;
>> > (see dftmagnitude3.zip attached in my previous mail and this one).
>> >
>> > So things for you to check:
>> >   * does the simple dftmagnitude3.zip compile on your system?
>> >   * what version of SWIG are you using? (I used 1.3.36 provided with
>> > cygwin)
>> >   * do you have a %include statement somewhere in your own .i file?
>> >
>> > Matthieu, if you read this, there's a complete example provided in
>> > dftmagnitude3.zip.
>> >   * Wrapped function sms_spectrumMag in dftmagnitude.c and .h
>> >   * SWIG wrapper dftmagnitude.i uses %inline and %rename statements
>> >   * Example uses a modified numpy.i (see the previous mails in the
>> > thread).
>> >   * test example provided in test_dftmagnitude.py
>> >
>> > Haven't tested it under Linux, but under winxp/cygwin/mingw32, the
>> > following
>> > works for me (in cygwin):
>> >
>> > $ python setup_dftmagnitude.py build -cmingw32 ; mv
>> > build/lib.win32-2.5/_dftmagnitude.pyd .
>> > $ python test_dftmagnitude.py
>> >
>> > Regards,
>> > Egor
>> >
>> > --
>> > My Python:
>> > $ python -i
>> > Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
>> > (Intel)]
>> > on win32
>> >
>> > My SWIG:
>> > $ swig -version
>> >
>> > SWIG Version 1.3.36
>> >
>> > Compiled with g++ [i686-pc-cygwin]
>> > Please see http://www.swig.org for reporting bugs and further
>> > information
>> >
>> >
>> >
>> >
>> > On Thu, Jan 8, 2009 at 1:43 AM, Rich E <reakinator at gmail.com> wrote:
>> >>
>> >> Here is my example, trying to wrap the function sms_spectrumMag that
>> >> we have been dealing with:
>> >>
>> >> %apply (int DIM1, float* IN_ARRAY1) {(int sizeInArray, float*
>> >> pInArray)};
>> >> %apply (int DIM1, float* INPLACE_ARRAY1) {(int sizeOutArray, float*
>> >> pOutArray)};
>> >>
>> >> %inline %{
>> >>
>> >>        void my_spectrumMag( int sizeInArray, float *pInArray, int
>> >> sizeOutArray, float *pOutArray)
>> >>        {
>> >>                sms_spectrumMag(sizeOutArray, pInArray, pOutArray);
>> >>        }
>> >>
>> >> %}
>> >>
>> >>
>> >> at this point,  have the new function my_spectrumMag that wraps
>> >> sms_spectrumMag() and provides arguments that can be typemapped using
>> >> numpy.i  Now, I don't want to have to call the function
>> >> my_spectrumMag() in python, I want to use the original name, I would
>> >> like to call the function as:
>> >>
>> >> sms_spectrumMag(numpyArray1, numpyArray2)
>> >>
>> >> But, trying to %rename my_spectrumMag to sms_spectrumMag does not
>> >> work, the original sms_spectrumMag gets called in python instead.
>> >> Trying to %ignore the original function first as follows removes the
>> >> sms_spectrumMag completely from the module and I am left with
>> >> my_spectrumMag:
>> >>
>> >> %ignore sms_spectrumMag;
>> >> %rename (sms_spectrumMag) my_spectrumMag;
>> >>
>> >>
>> >> Do you see my problem?
>> >>
>> >>
>> >> On Wed, Jan 7, 2009 at 8:58 AM, Matthieu Brucher
>> >> <matthieu.brucher at gmail.com> wrote:
>> >> > 2009/1/6 Rich E <reakinator at gmail.com>:
>> >> >> This helped immensely.  I feel like I am getting close to being able
>> >> >> to accomplish what I would like with SWIG: producing a python module
>> >> >> that can be very 'python-like', while co-existing with the c library
>> >> >> that is very 'c-like'.
>> >> >>
>> >> >> There is one question still remaining though, is it possible to make
>> >> >> the wrapped function have the same name still?  Using either
>> >> >> my_spectrumMag or spectrumMag means I have to create a number of
>> >> >> inconsistencies between the python module and the c library.  It is
>> >> >> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use the
>> >> >> wrapped one, with the same name.  But my attempts at doing this so
>> >> >> far
>> >> >> have not compiled because of name conflictions.
>> >> >
>> >> > Ok course you can. The function is renamed only if you say so.
>> >> > Perhaps
>> >> > can you provide a small example of what doesn't work at the moment ?
>> >> >
>> >> >> Thanks for the help, I think you are doing great things with this
>> >> >> numpy interface/typemaps system.
>> >> >
>> >> > Matthieu
>> >> > --
>> >> > Information System Engineer, Ph.D.
>> >> > Website: http://matthieu-brucher.developpez.com/
>> >> > Blogs: http://matt.eifelle.com and
>> >> > http://blog.developpez.com/?blog=92
>> >> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> >> > _______________________________________________
>> >> > Numpy-discussion mailing list
>> >> > Numpy-discussion at scipy.org
>> >> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> >> >
>> >> _______________________________________________
>> >> Numpy-discussion mailing list
>> >> Numpy-discussion at scipy.org
>> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>> > _______________________________________________
>> > Numpy-discussion mailing list
>> > Numpy-discussion at scipy.org
>> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>
>


From robert.kern at gmail.com  Fri Jan  9 17:12:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 9 Jan 2009 16:12:02 -0600
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4967C9F5.8070704@noaa.gov>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
	<3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
	<4967C449.9060508@noaa.gov>
	<b694391a022f8a0d37ea004e5fcc3a5a.squirrel@webmail.uio.no>
	<4967C9F5.8070704@noaa.gov>
Message-ID: <3d375d730901091412j3d82867dn85d21ab4e9da0818@mail.gmail.com>

On Fri, Jan 9, 2009 at 16:04, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Sturla Molden wrote:
>> For the same problems where you would use meshgrid in Matlab.
>
> well, I used to use meshgrid a lot because MATLAB could not do
> broadcasting. Which is probably why the OP has been trying to use it.
>
> A note for the docs:
>
> The docs refer to ogrid and nd_grid, and as far as I can tell, they are
> the same thing, but it's confusing. actuall, as I look more, ogrid is a
> nd_grid with the sparse parameter set to True -- anyway, still confusing!

I'm not sure why. The docstring seems fairly clear about this.

> Also, what is the "o" for -- the mnemonic might be helpful

"open". mgrid and ogrid are instantiations of the nd_grid class with
different parameters.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From sturla at molden.no  Fri Jan  9 17:27:14 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 9 Jan 2009 23:27:14 +0100 (CET)
Subject: [Numpy-discussion] Numpy performance vs Matlab.
In-Reply-To: <4967C9F5.8070704@noaa.gov>
References: <8c051b18e6c219c05fcc3c126f1d566d.squirrel@webmail.uio.no>
	<015501c97280$4c036100$e7ad810a@gnb.st.com>
	<3d375d730901091320x5bab1fabs9bfe740f10914f08@mail.gmail.com>
	<4967C449.9060508@noaa.gov>
	<b694391a022f8a0d37ea004e5fcc3a5a.squirrel@webmail.uio.no>
	<4967C9F5.8070704@noaa.gov>
Message-ID: <142c3d71d99a9a9c8e22c174e3786c0a.squirrel@webmail.uio.no>

> Sturla Molden wrote:
>> For the same problems where you would use meshgrid in Matlab.
>
> well, I used to use meshgrid a lot because MATLAB could not do
> broadcasting. Which is probably why the OP has been trying to use it.

mgrid and ogrid are both meshgrids, with ogrid having a sparse
representation (i.e. it uses less memory).

There was no need for broadcasting in the OP's code. It just seemed a bit
confused.

Regards,
Sturla Molden





From pjssilva at ime.usp.br  Fri Jan  9 21:10:37 2009
From: pjssilva at ime.usp.br (Paulo J. S. Silva)
Date: Fri, 09 Jan 2009 23:10:37 -0300
Subject: [Numpy-discussion] 2-D function and meshgrid
In-Reply-To: <e06186140901091237q1d680d3sb5fd06b0924def52@mail.gmail.com>
References: <1231526719.28818.19.camel@trinity>
	<e06186140901091237q1d680d3sb5fd06b0924def52@mail.gmail.com>
Message-ID: <1231553437.10173.27.camel@trinity>

Chuck,

Thanks, your version is much faster. I would prefer a solution that
doesn't force me to re-implement weirdDistance (as my two solutions
were). But the function is so simple that it is easier just to re-write
it for speed as you did.

By the way, I came out with one more solution that looks more Pythonic
and does not need to re-write weirdDistance (and hence can be used in
more complicated cases). It is also a tad faster than the fastest
solution from my first post:

Solution 3

    points = np.vstack( [x.ravel(), y.ravel()] ).T
    results = np.array([weirDistance(p) for p in points])
    return results.reshape(x.shape)

(This is basically solution 2 using list comprehensions to make to code
clearer)

best,

Paulo
> 
> Try
> 
> IDLE 1.2.4      
> >>> import numpy as np
> >>> pts = np.random.rand(5,2)
> >>> mat = np.random.rand(2,2)
> >>> res = (np.dot(pts,mat)*pts).sum(axis=1)
> >>> res
> 
> array([ 0.63018561,  0.30829864,  0.23173343,  1.79972127,
> 0.69498856])
> >>> for row in pts : np.dot(row,np.dot(mat,row))
> 
> 0.63018560596590589
> 0.30829864146737423
> 0.23173343333294744
> 1.7997212735553192
> 0.69498855520540959
> 
> Chuck
> 





From ezindy at gmail.com  Fri Jan  9 21:29:52 2009
From: ezindy at gmail.com (Egor Zindy)
Date: Sat, 10 Jan 2009 11:29:52 +0900
Subject: [Numpy-discussion] help with typemapping a C function to use
	numpy arrays
In-Reply-To: <a3297bc90901091411w670158aas404cf14fb8d9f577@mail.gmail.com>
References: <a3297bc90812230852v6c19e7deo9f0da671c1c70259@mail.gmail.com>
	<a3297bc90901050806w50b84875sd574901a236451c8@mail.gmail.com>
	<c156de00901060647t442503ddsef7516348067fa9@mail.gmail.com>
	<a3297bc90901061451x115aceaap9e2c9db4f3489508@mail.gmail.com>
	<e76aa17f0901062358v16c69fact241b8206f849f588@mail.gmail.com>
	<a3297bc90901070843s734f8224pe4724afcdc03af3a@mail.gmail.com>
	<c156de00901071713i31876ec3u84d39b2ac4f27ab3@mail.gmail.com>
	<a3297bc90901081243g3c04e089i94417f7d3438d3a5@mail.gmail.com>
	<c156de00901082205q2149955jf5cba8ea987c9a3@mail.gmail.com>
	<a3297bc90901091411w670158aas404cf14fb8d9f577@mail.gmail.com>
Message-ID: <c156de00901091829v10a02d87rdd04b10ccc81284e@mail.gmail.com>

Hello again!

On Sat, Jan 10, 2009 at 7:11 AM, Rich E <reakinator at gmail.com> wrote:

> Well I see it works, however with one change: the %apply typemaps need
> to be done before %include'ing the header file, or else nothing in
> that header file will automatically get typemapped (only the functions
> that are written using %inline will be typemapped, which in the case
> of the exampe you wrote, is all there is.  This isn't the case in the
> library I am writing, there are about 20-30 others that don't need to
> be written inline).


There were no typemaps in the example you gave, so it's nice to know you
found where to put the typemaps definition lines in respect to everything
else (this should definitely go in the cookbook!)


>
> One comment though, upon looking at the wrapped c file: there are two
> definitions for sms_spectrumMag(), one that expects pointers to c
> arrays as arguments, and then one later that expects numpy arrays as
> arguments.


I just had a look at the dftmagnitude_wrap.c and I'm confused. If you
understand what's going on, maybe you can shed some light?

The definition for python is contained here:

static PyMethodDef SwigMethods[] = {
     { (char *)"sms_spectrumMag", _wrap_sms_spectrumMag, METH_VARARGS,
NULL},
     { NULL, NULL, 0, NULL }
};

function name on the python side is "sms_spectrumMag"

The _wrap_sms_spectrumMag function the calls either
_wrap_sms_spectrumMag__SWIG_1 (argc == 1)
or
_wrap_sms_spectrumMag__SWIG_0 (argc == 3)

depending on argc = (int)PyObject_Length(args);

.... This is where things become complicated...

_wrap_sms_spectrumMag__SWIG_1 calls my_spectrumMag, the inlined function,

but _wrap_sms_spectrumMag__SWIG_0 doesn't! _wrap_sms_spectrumMag__SWIG_0
calls sms_spectrumMag directly with the 3 arguments converted from python:
int, float * and float *!

Isn't this what we were trying to avoid in the first place? Is there any way
to instruct SWIG not to wrap the function directly?


>  Upon testing the function in python, it seems there is no
> conflict; supplying just a numpy array works perfectly fine.  I dont'
> understand how python is interpreting this so I cannot foresee any
> problems, I'm just bringing it up in case others can.
>

I don't understand either. I thought I did, though! My understanding is that
the function name in python is just a string and can be anything instructed
by the %rename statement. What I don't understand is that SWIG still tries
to wrap the original function, even though we are only interested in
wrapping the %inline-d one.


>
> So now I can wrap the functions in a way where they are very useful in
> python without messing up the c code.


Yup, if you call the python function with the intended number of arguments
(and don't end-up calling the wrapped original function).



>  Great, thank you Egor for the
> ongoing help!  Sorry it took your vacation, but you helped me spend
> mine the way I wanted to (successful programming, hehe).


Don't mention it! I call that a "productive" holiday ;-)

Regards,
Egor


>
> cheers,
> Rich
>
> On Fri, Jan 9, 2009 at 7:05 AM, Egor Zindy <ezindy at gmail.com> wrote:
> > Hello Rich,
> >
> > I know what you mean. %inclusion of header files saves a lot of effort!
> So,
> > I had another play with the code (what holiday this turned out to be;)
> and
> > as long as the declarations in the .i file are made in the right order,
> it
> > should be possible to:
> >   * %include the header file
> >   * %ignore a sms_ function
> >   * %rename the function my_ to sms_
> >   * %inline the my_ function
> >
> > I changed the .i file (attached) and re-ran the test, it works. Again,
> this
> > is on my XP/cygwin/mingw32 system, so it could need some tuning on a
> > different system!
> >
> > In all this, not sure where is best to put the %exception statement, but
> > placement shouldn't be critical, because it concerns the my_ function
> rather
> > than the original (or renamed) sms_ function.
> >
> > Regards,
> > Egor
> >
> > On Fri, Jan 9, 2009 at 5:43 AM, Rich E <reakinator at gmail.com> wrote:
> >>
> >> I am using %includ "sms.h", which is what is wrapping all my
> >> functions.  Without doing this, I have to hand-wrap every function in
> >> the header file!
> >>
> >> Is there a way to exclude certain definitions from my c header file
> >> when using %include, so that I can hand wrap them instead?
> >>
> >> On Thu, Jan 8, 2009 at 2:13 AM, Egor Zindy <ezindy at gmail.com> wrote:
> >> > Hello Rich,
> >> >
> >> > This is very strange. I got to test my example again, as long as you
> >> > don't
> >> > do a
> >> > %include "dftmagnitude.h"
> >> > somewhere in the dftmagnitude.i, it's perfectly possible to do a
> >> > %rename (sms_spectrumMag) my_spectrumMag;
> >> > (see dftmagnitude3.zip attached in my previous mail and this one).
> >> >
> >> > So things for you to check:
> >> >   * does the simple dftmagnitude3.zip compile on your system?
> >> >   * what version of SWIG are you using? (I used 1.3.36 provided with
> >> > cygwin)
> >> >   * do you have a %include statement somewhere in your own .i file?
> >> >
> >> > Matthieu, if you read this, there's a complete example provided in
> >> > dftmagnitude3.zip.
> >> >   * Wrapped function sms_spectrumMag in dftmagnitude.c and .h
> >> >   * SWIG wrapper dftmagnitude.i uses %inline and %rename statements
> >> >   * Example uses a modified numpy.i (see the previous mails in the
> >> > thread).
> >> >   * test example provided in test_dftmagnitude.py
> >> >
> >> > Haven't tested it under Linux, but under winxp/cygwin/mingw32, the
> >> > following
> >> > works for me (in cygwin):
> >> >
> >> > $ python setup_dftmagnitude.py build -cmingw32 ; mv
> >> > build/lib.win32-2.5/_dftmagnitude.pyd .
> >> > $ python test_dftmagnitude.py
> >> >
> >> > Regards,
> >> > Egor
> >> >
> >> > --
> >> > My Python:
> >> > $ python -i
> >> > Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
> >> > (Intel)]
> >> > on win32
> >> >
> >> > My SWIG:
> >> > $ swig -version
> >> >
> >> > SWIG Version 1.3.36
> >> >
> >> > Compiled with g++ [i686-pc-cygwin]
> >> > Please see http://www.swig.org for reporting bugs and further
> >> > information
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Jan 8, 2009 at 1:43 AM, Rich E <reakinator at gmail.com> wrote:
> >> >>
> >> >> Here is my example, trying to wrap the function sms_spectrumMag that
> >> >> we have been dealing with:
> >> >>
> >> >> %apply (int DIM1, float* IN_ARRAY1) {(int sizeInArray, float*
> >> >> pInArray)};
> >> >> %apply (int DIM1, float* INPLACE_ARRAY1) {(int sizeOutArray, float*
> >> >> pOutArray)};
> >> >>
> >> >> %inline %{
> >> >>
> >> >>        void my_spectrumMag( int sizeInArray, float *pInArray, int
> >> >> sizeOutArray, float *pOutArray)
> >> >>        {
> >> >>                sms_spectrumMag(sizeOutArray, pInArray, pOutArray);
> >> >>        }
> >> >>
> >> >> %}
> >> >>
> >> >>
> >> >> at this point,  have the new function my_spectrumMag that wraps
> >> >> sms_spectrumMag() and provides arguments that can be typemapped using
> >> >> numpy.i  Now, I don't want to have to call the function
> >> >> my_spectrumMag() in python, I want to use the original name, I would
> >> >> like to call the function as:
> >> >>
> >> >> sms_spectrumMag(numpyArray1, numpyArray2)
> >> >>
> >> >> But, trying to %rename my_spectrumMag to sms_spectrumMag does not
> >> >> work, the original sms_spectrumMag gets called in python instead.
> >> >> Trying to %ignore the original function first as follows removes the
> >> >> sms_spectrumMag completely from the module and I am left with
> >> >> my_spectrumMag:
> >> >>
> >> >> %ignore sms_spectrumMag;
> >> >> %rename (sms_spectrumMag) my_spectrumMag;
> >> >>
> >> >>
> >> >> Do you see my problem?
> >> >>
> >> >>
> >> >> On Wed, Jan 7, 2009 at 8:58 AM, Matthieu Brucher
> >> >> <matthieu.brucher at gmail.com> wrote:
> >> >> > 2009/1/6 Rich E <reakinator at gmail.com>:
> >> >> >> This helped immensely.  I feel like I am getting close to being
> able
> >> >> >> to accomplish what I would like with SWIG: producing a python
> module
> >> >> >> that can be very 'python-like', while co-existing with the c
> library
> >> >> >> that is very 'c-like'.
> >> >> >>
> >> >> >> There is one question still remaining though, is it possible to
> make
> >> >> >> the wrapped function have the same name still?  Using either
> >> >> >> my_spectrumMag or spectrumMag means I have to create a number of
> >> >> >> inconsistencies between the python module and the c library.  It
> is
> >> >> >> ideal to ignore (%ignore?) the c sms_spectrumMag and instead use
> the
> >> >> >> wrapped one, with the same name.  But my attempts at doing this so
> >> >> >> far
> >> >> >> have not compiled because of name conflictions.
> >> >> >
> >> >> > Ok course you can. The function is renamed only if you say so.
> >> >> > Perhaps
> >> >> > can you provide a small example of what doesn't work at the moment
> ?
> >> >> >
> >> >> >> Thanks for the help, I think you are doing great things with this
> >> >> >> numpy interface/typemaps system.
> >> >> >
> >> >> > Matthieu
> >> >> > --
> >> >> > Information System Engineer, Ph.D.
> >> >> > Website: http://matthieu-brucher.developpez.com/
> >> >> > Blogs: http://matt.eifelle.com and
> >> >> > http://blog.developpez.com/?blog=92
> >> >> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> >> >> > _______________________________________________
> >> >> > Numpy-discussion mailing list
> >> >> > Numpy-discussion at scipy.org
> >> >> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >> >> >
> >> >> _______________________________________________
> >> >> Numpy-discussion mailing list
> >> >> Numpy-discussion at scipy.org
> >> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >> >
> >> >
> >> > _______________________________________________
> >> > Numpy-discussion mailing list
> >> > Numpy-discussion at scipy.org
> >> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090110/1340fe07/attachment.html>

From ndbecker2 at gmail.com  Fri Jan  9 23:15:27 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 09 Jan 2009 23:15:27 -0500
Subject: [Numpy-discussion] memmap from fd?
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
	<gk7lpr$q9j$1@ger.gmane.org>
	<3d375d730901091310v27033d5co1fbdc71098b9e672@mail.gmail.com>
Message-ID: <gk97cv$egf$1@ger.gmane.org>

Robert Kern wrote:

> On Fri, Jan 9, 2009 at 08:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>> Robert Kern wrote:
>>
>>> On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>> I'm working on interfacing to a custom FPGA board.  The kernel driver
>>>> exposes the FPGA memory via mmap.
>>>>
>>>> It might be nice to use numpy memmap to read/write data.  One issue is
>>>> that I think I will need to create the memmap array from a fd, not a
>>>> file
>>>> name.  The reason is I wrote the driver to only allow 1 exclusive open,
>>>> and I already have it open for other reasons.  Any chance to create a
>>>> memmap array from a fd?
>>>
>>> Use os.fdopen(fd) to create a file object which can be passed to the
>>> memmap constructor.
>>>
>>
>> Looks like this is not going to work without some change to memmap.  The
>> problem is, I need read/write access.  The only choice in memmap is 'w+'.
> 
> 'r+' is for reading and writing.
> 
>> But this does:
>>        if (mode == 'w+') and shape is None:
>>            raise ValueError, "shape must be given"
>>
>>        fid.seek(0,2)
>>
>> My device has hijacked 'read' to mean something entirely different than
>> you might expect.  The seek call invokes 'read'.
>>
>> It looks like the purpose of this code is to find the size of the
>> mappable area.  The best solution I think is just throw it away.
> 
> We can't. We need it.
> 
>> Consistent with mmap semantics, attempting access outside the mappable
>> area should cause and error - but I don't think there is any reliable way
>> to know the length of the mappable area apriori.
> 
> For regular files, that seems to me to be fairly reliable. Why isn't it?
> 
Because I'm not mmapping a file.  I'm mmapping a device.  It exposes the memory of the FPGA board as seen on the PCI bus.  You just have to know the size.



From ndbecker2 at gmail.com  Sat Jan 10 13:41:59 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sat, 10 Jan 2009 13:41:59 -0500
Subject: [Numpy-discussion] memmap problem
References: <gk7vqf$1uq$1@ger.gmane.org>
	<3d375d730901091253n597a294au13d9c32ef76463e5@mail.gmail.com>
Message-ID: <gkaq5n$du6$1@ger.gmane.org>

Robert Kern wrote:

> On Fri, Jan 9, 2009 at 10:59, Neal Becker <ndbecker2 at gmail.com> wrote:
>> I modified memmap.py to avoid the issues with needed to read.  It is
>> working, but I am seeing these:
>>
>>  m
>> Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound
>> method eos_memmap.__del__ of eos_memmap([255, 255, 255], dtype=uint8)>
>> ignored Exception exceptions.EnvironmentError: (22, 'Invalid argument')
>> in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0],
>> dtype=uint8)> ignored
>> Out[22]: eos_memmap([  0,   0,   0, ..., 255, 255, 255], dtype=uint8)
>>
>> print m[0:4]
>> Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound
>> method eos_memmap.__del__ of eos_memmap([0, 0, 0, 0], dtype=uint8)>
>> ignored
>> [0 0 0 0]
>>
>> What's that about?
> 
> Can you show us your modifications? You may need to also modify
> __array_finalize__(), _close(), and __del__().
> 

Problem was with __del__ calling flush.  For whatever reason, flush is bad on my device.  I simply removed the call.



From robert.kern at gmail.com  Sat Jan 10 20:41:47 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 10 Jan 2009 19:41:47 -0600
Subject: [Numpy-discussion] memmap from fd?
In-Reply-To: <gk97cv$egf$1@ger.gmane.org>
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
	<gk7lpr$q9j$1@ger.gmane.org>
	<3d375d730901091310v27033d5co1fbdc71098b9e672@mail.gmail.com>
	<gk97cv$egf$1@ger.gmane.org>
Message-ID: <3d375d730901101741p22facd24qe6b391a0b023bd6f@mail.gmail.com>

On Fri, Jan 9, 2009 at 22:15, Neal Becker <ndbecker2 at gmail.com> wrote:
> Robert Kern wrote:
>
>> On Fri, Jan 9, 2009 at 08:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> Robert Kern wrote:
>>>
>>>> On Fri, Jan 9, 2009 at 06:05, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>> I'm working on interfacing to a custom FPGA board.  The kernel driver
>>>>> exposes the FPGA memory via mmap.
>>>>>
>>>>> It might be nice to use numpy memmap to read/write data.  One issue is
>>>>> that I think I will need to create the memmap array from a fd, not a
>>>>> file
>>>>> name.  The reason is I wrote the driver to only allow 1 exclusive open,
>>>>> and I already have it open for other reasons.  Any chance to create a
>>>>> memmap array from a fd?
>>>>
>>>> Use os.fdopen(fd) to create a file object which can be passed to the
>>>> memmap constructor.
>>>>
>>>
>>> Looks like this is not going to work without some change to memmap.  The
>>> problem is, I need read/write access.  The only choice in memmap is 'w+'.
>>
>> 'r+' is for reading and writing.
>>
>>> But this does:
>>>        if (mode == 'w+') and shape is None:
>>>            raise ValueError, "shape must be given"
>>>
>>>        fid.seek(0,2)
>>>
>>> My device has hijacked 'read' to mean something entirely different than
>>> you might expect.  The seek call invokes 'read'.
>>>
>>> It looks like the purpose of this code is to find the size of the
>>> mappable area.  The best solution I think is just throw it away.
>>
>> We can't. We need it.
>>
>>> Consistent with mmap semantics, attempting access outside the mappable
>>> area should cause and error - but I don't think there is any reliable way
>>> to know the length of the mappable area apriori.
>>
>> For regular files, that seems to me to be fairly reliable. Why isn't it?
>>
> Because I'm not mmapping a file.  I'm mmapping a device.  It exposes the memory of the FPGA board as seen on the PCI bus.  You just have to know the size.

That's a fair point. However, given the wider range of issues that you
had in trying to get memmap to work with your device, perhaps we
should just state that the memmap class is intended for the common use
case of regular files. The detection of the existing file size is
quite useful in that case. For special files where seek() et al. might
have different semantics, one should either subclass memmap to get the
behavior you need, or simply use frombuffer() or ndarray() on the mmap
object directly. The two reasons for the memmap class are the
auto-length-detection and tracking the lifetime of the open file. The
former you don't need (and can't do), and the latter isn't that hard
to do yourself.

Does that sound reasonable?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bryan at cole.uklinux.net  Sun Jan 11 04:41:31 2009
From: bryan at cole.uklinux.net (Bryan Cole)
Date: Sun, 11 Jan 2009 09:41:31 +0000
Subject: [Numpy-discussion] TypeError in dtype.__eq__()
Message-ID: <1231666889.11536.42.camel@pc2.cole.uklinux.net>

Dtype objects throw an exception if compared for equality against other
objects. e.g.

>>> import numpy
>>> numpy.dtype('uint32')==1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: data type not understood
>>> 

After some googling, I think python wisdom (given in the Python docs)
says that equality testing should never throw an exception, but return
the NotImplemented singleton where the comparison does not make "sense".
Throwing an exception breaks membership testing for lists/tuples (using
the 'in' operator calls implicitly calls __eq__()). 

My real-world example is serialisation of numpy arrays using PyYAML.
This module should serialise any picklable object including ndarrays.
However, it performs a test on each object:

    if data in [None, ()]: blah...

This fails on the dtype object of the array.

What's the consensus on this? Is the current dtype behaviour broken?

cheers,
BC




From robert.kern at gmail.com  Sun Jan 11 04:49:11 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 11 Jan 2009 03:49:11 -0600
Subject: [Numpy-discussion] TypeError in dtype.__eq__()
In-Reply-To: <1231666889.11536.42.camel@pc2.cole.uklinux.net>
References: <1231666889.11536.42.camel@pc2.cole.uklinux.net>
Message-ID: <3d375d730901110149l4d8cf182raa1f9af346849f50@mail.gmail.com>

On Sun, Jan 11, 2009 at 03:41, Bryan Cole <bryan at cole.uklinux.net> wrote:
> Dtype objects throw an exception if compared for equality against other
> objects. e.g.
>
>>>> import numpy
>>>> numpy.dtype('uint32')==1
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: data type not understood
>>>>
>
> After some googling, I think python wisdom (given in the Python docs)
> says that equality testing should never throw an exception, but return
> the NotImplemented singleton where the comparison does not make "sense".
> Throwing an exception breaks membership testing for lists/tuples (using
> the 'in' operator calls implicitly calls __eq__()).
>
> My real-world example is serialisation of numpy arrays using PyYAML.
> This module should serialise any picklable object including ndarrays.
> However, it performs a test on each object:
>
>    if data in [None, ()]: blah...
>
> This fails on the dtype object of the array.
>
> What's the consensus on this? Is the current dtype behaviour broken?

It's suboptimal, certainly. Feel free to fix it. However, also note
that with ndarray's rich comparisons, such membership testing will
fail with ndarrays, too.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bryan at cole.uklinux.net  Sun Jan 11 11:19:51 2009
From: bryan at cole.uklinux.net (Bryan Cole)
Date: Sun, 11 Jan 2009 16:19:51 +0000
Subject: [Numpy-discussion] TypeError in dtype.__eq__()
In-Reply-To: <3d375d730901110149l4d8cf182raa1f9af346849f50@mail.gmail.com>
References: <1231666889.11536.42.camel@pc2.cole.uklinux.net>
	<3d375d730901110149l4d8cf182raa1f9af346849f50@mail.gmail.com>
Message-ID: <1231690790.11536.64.camel@pc2.cole.uklinux.net>


> >
> > What's the consensus on this? Is the current dtype behaviour broken?
> 
> It's suboptimal, certainly. Feel free to fix it. 

Thankyou. 

> However, also note
> that with ndarray's rich comparisons, such membership testing will
> fail with ndarrays, too.

This poses a similarly big problem. I can't understand this behaviour
either:

>>> a = numpy.array([1.2,3.4])
>>> a == 6.5
array([False, False], dtype=bool)
>>> a == numpy.array([1,2])
array([False, False], dtype=bool)
>>> a == "foo"
False
>>> a in [1,2,3]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

Surely this is a bug?! This membership testing operation is quite
reasonable. If list.__contains__ is calling ndarray.__eq__(), this ought
to succeed. Why does this fail? (and I fail to see how any() or all()
can resolve the situation).

I need to go check what list.__contains__() is doing...

BC




From aisaac at american.edu  Sun Jan 11 15:55:28 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 11 Jan 2009 15:55:28 -0500
Subject: [Numpy-discussion] coding style: citations
Message-ID: <496A5CC0.8050605@american.edu>

The docstring standard at
http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines#docstring-standard
suggests a citation reference format that is not compatible
with reStructuredText.  Quoting from
http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#citations:

	Citations are identical to footnotes except that they use
         only non-numeric labels such as [note] or [GVR2001].

In other words, the standard currently promotes a conflation of
notes and citations.

Another reason for avoiding the style currently in the docstring
standard is that it renders impossible the generation of citations
using a citation database.  (See e.g. the bibtext-like facilities offered
by bibstuff's bib4txt (http://code.google.com/p/bibstuff/).

Alan Isaac



From stefan at sun.ac.za  Sun Jan 11 16:13:01 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 11 Jan 2009 23:13:01 +0200
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496A5CC0.8050605@american.edu>
References: <496A5CC0.8050605@american.edu>
Message-ID: <9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>

Hi Alan

2009/1/11 Alan G Isaac <aisaac at american.edu>:
> The docstring standard at
> http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines#docstring-standard
> suggests a citation reference format that is not compatible
> with reStructuredText.  Quoting from
> http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#citations:
>
>        Citations are identical to footnotes except that they use
>         only non-numeric labels such as [note] or [GVR2001].
>
> In other words, the standard currently promotes a conflation of
> notes and citations.

Thank you for your feedback.  Yes, this is a problem.  In a way,
RestructuredText is partially to blame for not providing  numerical
citation markup.  If we can come up with a good way of generating the
reference keys, we can still change the current format.

Maybe the library you mention below can automatically generate such
keys?  I'd like to take that responsibility away from documentation
writers, since it is not always obvious how to generate consistent
keys.

Regards
St?fan




>
> Another reason for avoiding the style currently in the docstring
> standard is that it renders impossible the generation of citations
> using a citation database.  (See e.g. the bibtext-like facilities offered
> by bibstuff's bib4txt (http://code.google.com/p/bibstuff/).
>
> Alan Isaac
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From bryan at cole.uklinux.net  Sun Jan 11 16:13:24 2009
From: bryan at cole.uklinux.net (Bryan Cole)
Date: Sun, 11 Jan 2009 21:13:24 +0000
Subject: [Numpy-discussion] TypeError in dtype.__eq__()
In-Reply-To: <1231690790.11536.64.camel@pc2.cole.uklinux.net>
References: <1231666889.11536.42.camel@pc2.cole.uklinux.net>
	<3d375d730901110149l4d8cf182raa1f9af346849f50@mail.gmail.com>
	<1231690790.11536.64.camel@pc2.cole.uklinux.net>
Message-ID: <1231708403.11536.99.camel@pc2.cole.uklinux.net>


> > However, also note
> > that with ndarray's rich comparisons, such membership testing will
> > fail with ndarrays, too.
> 
> This poses a similarly big problem. I can't understand this behaviour
> either:

OK, I can now. After equality testing each item, the result must be cast
to bool. This is where the truth-testing comes in.

Testing array membership in lists does work, just so long as there are
no numerical objects in the list. Any/only equality comparisons which
return another (bool) ndarray will cause the exception. Thus, the error
is not triggered in PyYAML.

FWIW, I don't like the policy of forbidding truth testing on bare
arrays. I'd prefer that ndarray.__nonzero__() implicitly call
ndarray.all(). For my use-cases, this is what I want >90% of the time.
Where any() is required, this must be made explicit. However, I
understand the logic behind raising an exception and presumably others
prefer this.

Ultimately, I'm forced to conclude that the idea that membership testing
should always succeed was bogus. Bummer.

BC




From aisaac at american.edu  Sun Jan 11 17:05:10 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 11 Jan 2009 17:05:10 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>
References: <496A5CC0.8050605@american.edu>
	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>
Message-ID: <496A6D16.5050501@american.edu>

On 1/11/2009 4:13 PM St?fan van der Walt apparently wrote:
> Thank you for your feedback.  Yes, this is a problem.  In a way,
> RestructuredText is partially to blame for not providing  numerical
> citation markup.

I do not agree.

I cannot think of any bibliography tool that uses
numerical citation reference keys. Naturally there
are many that *substitute* numerical citation references
for the cite key used by the author.  This is a *style*
question that should be automated (i.e., not be a
responsibility of the author).  E.g., LaTeX provides
a variety of styles for cite key substitution.

> If we can come up with a good way of generating the
> reference keys, we can still change the current format.

This would really involve the following.
Create a searchable database of citations
and an interface for adding to it.
Unique keys would be generated by your
algorithm of choice when an entry is added.
Authors would be asked to use only references
in the database.

Desirable for a book.  Desirable for documentation?

> Maybe the library you mention below can automatically generate such
> keys?  I'd like to take that responsibility away from documentation
> writers, since it is not always obvious how to generate consistent
> keys.

Numerical keys will clearly *not* be consistent.
The same key will refer to different citations
on different pages, and key width will not be
uniform. In additional, numerical keys are
not informative when encountered by the reader.
I would prefer [last1.last2-2009-sja]_ where
sja is "standard journal abbreviation" and
last names are ASCII (e.g., ? -> o).
All lower case. This is very informative
and easy for all users. It also means the
key is valid for both HTML and XML uses
(e.g., as a name or id).

But to answer your question, bibstuff includes
biblabel.py, which can produce keys for a bibtex
database (styled as you like).  The problem of
setting up the data base remains.

Alan Isaac



From robert.kern at gmail.com  Sun Jan 11 19:48:11 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 11 Jan 2009 18:48:11 -0600
Subject: [Numpy-discussion] TypeError in dtype.__eq__()
In-Reply-To: <1231708403.11536.99.camel@pc2.cole.uklinux.net>
References: <1231666889.11536.42.camel@pc2.cole.uklinux.net>
	<3d375d730901110149l4d8cf182raa1f9af346849f50@mail.gmail.com>
	<1231690790.11536.64.camel@pc2.cole.uklinux.net>
	<1231708403.11536.99.camel@pc2.cole.uklinux.net>
Message-ID: <3d375d730901111648o7f86a040g51381fe7ea77525f@mail.gmail.com>

On Sun, Jan 11, 2009 at 15:13, Bryan Cole <bryan at cole.uklinux.net> wrote:
>
>> > However, also note
>> > that with ndarray's rich comparisons, such membership testing will
>> > fail with ndarrays, too.
>>
>> This poses a similarly big problem. I can't understand this behaviour
>> either:
>
> OK, I can now. After equality testing each item, the result must be cast
> to bool. This is where the truth-testing comes in.
>
> Testing array membership in lists does work, just so long as there are
> no numerical objects in the list. Any/only equality comparisons which
> return another (bool) ndarray will cause the exception. Thus, the error
> is not triggered in PyYAML.
>
> FWIW, I don't like the policy of forbidding truth testing on bare
> arrays. I'd prefer that ndarray.__nonzero__() implicitly call
> ndarray.all(). For my use-cases, this is what I want >90% of the time.
> Where any() is required, this must be made explicit. However, I
> understand the logic behind raising an exception and presumably others
> prefer this.

That almost works for == but doesn't for !=. We used to use .any() in
Numeric (if any element is nonzero, then the array is nonzero). Many
people were confused. There was a lot of broken code out there. Moving
to numpy, we decided to prevent that kind of silently broken code.

In any case, it still wouldn't work. Because the argument are
broadcasted against each other, you can have two differently shaped
arrays compare equal. E.g. (ones([5,3]) == 1.0).all() == True.

> Ultimately, I'm forced to conclude that the idea that membership testing
> should always succeed was bogus. Bummer.

Yup. Sorry. It's an unfortunate effect of allowing rich comparisons.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Sun Jan 11 21:23:01 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 11 Jan 2009 21:23:01 -0500
Subject: [Numpy-discussion] memmap from fd?
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
	<gk7lpr$q9j$1@ger.gmane.org>
	<3d375d730901091310v27033d5co1fbdc71098b9e672@mail.gmail.com>
	<gk97cv$egf$1@ger.gmane.org>
	<3d375d730901101741p22facd24qe6b391a0b023bd6f@mail.gmail.com>
Message-ID: <gke9i6$th6$1@ger.gmane.org>

Robert Kern wrote:

....
> 
> That's a fair point. However, given the wider range of issues that you
> had in trying to get memmap to work with your device, perhaps we
> should just state that the memmap class is intended for the common use
> case of regular files. The detection of the existing file size is
> quite useful in that case. For special files where seek() et al. might
> have different semantics, one should either subclass memmap to get the
> behavior you need, or simply use frombuffer() or ndarray() on the mmap
> object directly. The two reasons for the memmap class are the
> auto-length-detection and tracking the lifetime of the open file. The
> former you don't need (and can't do), and the latter isn't that hard
> to do yourself.
> 
> Does that sound reasonable?
> 

Hmm.  frombuffer sounds nice, but python mmap doesn't expose buffer interface (I just added a feature request for this).  I suppose I could write my own mmap module.

I couldn't find the reference to ndarray().  Any pointer?



From robert.kern at gmail.com  Sun Jan 11 21:45:39 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 11 Jan 2009 20:45:39 -0600
Subject: [Numpy-discussion] memmap from fd?
In-Reply-To: <gke9i6$th6$1@ger.gmane.org>
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
	<gk7lpr$q9j$1@ger.gmane.org>
	<3d375d730901091310v27033d5co1fbdc71098b9e672@mail.gmail.com>
	<gk97cv$egf$1@ger.gmane.org>
	<3d375d730901101741p22facd24qe6b391a0b023bd6f@mail.gmail.com>
	<gke9i6$th6$1@ger.gmane.org>
Message-ID: <3d375d730901111845vf02f364g87f78dbeeea292bf@mail.gmail.com>

On Sun, Jan 11, 2009 at 20:23, Neal Becker <ndbecker2 at gmail.com> wrote:
> Robert Kern wrote:
>
> ....
>>
>> That's a fair point. However, given the wider range of issues that you
>> had in trying to get memmap to work with your device, perhaps we
>> should just state that the memmap class is intended for the common use
>> case of regular files. The detection of the existing file size is
>> quite useful in that case. For special files where seek() et al. might
>> have different semantics, one should either subclass memmap to get the
>> behavior you need, or simply use frombuffer() or ndarray() on the mmap
>> object directly. The two reasons for the memmap class are the
>> auto-length-detection and tracking the lifetime of the open file. The
>> former you don't need (and can't do), and the latter isn't that hard
>> to do yourself.
>>
>> Does that sound reasonable?
>
> Hmm.  frombuffer sounds nice, but python mmap doesn't expose buffer interface (I just added a feature request for this).  I suppose I could write my own mmap module.

Yeah, it does. Why do you think it doesn't? If it didn't work, the
memmap class wouldn't work.

> I couldn't find the reference to ndarray().  Any pointer?

http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html

The buffer= argument would be the mmap object. Basically, most of what
memmap.__new__() does is to set up the arguments for
ndarray.__new__().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Mon Jan 12 02:35:15 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 12 Jan 2009 09:35:15 +0200
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496A6D16.5050501@american.edu>
References: <496A5CC0.8050605@american.edu>
	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>
	<496A6D16.5050501@american.edu>
Message-ID: <9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>

2009/1/12 Alan G Isaac <aisaac at american.edu>:
> This would really involve the following.
> Create a searchable database of citations
> and an interface for adding to it.
> Unique keys would be generated by your
> algorithm of choice when an entry is added.
> Authors would be asked to use only references
> in the database.
>
> Desirable for a book.  Desirable for documentation?

In documentation, you want the reference to appear in the docstring
itself.  Our docstrings double as the content of a book, which is why
it may be easier to extract the bibliography from the docstrings,
rather than populating the docstrings from a central bibliography.

> Numerical keys will clearly *not* be consistent.
> The same key will refer to different citations
> on different pages, and key width will not be
> uniform.

We automatically renumber the citations to take care of this.

> In additional, numerical keys are
> not informative when encountered by the reader.
> I would prefer [last1.last2-2009-sja]_ where
> sja is "standard journal abbreviation" and
> last names are ASCII (e.g., ? -> o).

I agree.

> But to answer your question, bibstuff includes
> biblabel.py, which can produce keys for a bibtex
> database (styled as you like).  The problem of
> setting up the data base remains.

We can add an interface to the documentation editor, where a person
pastes the BiBTeX reference, and it returns the appropriate key to use
in the docs.  Mabe you can think of a more intuitive interface, even.
As long as we have a consistent way of generating keys, I'd gladly use
them.

Regards
St?fan


From ndbecker2 at gmail.com  Mon Jan 12 08:10:55 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 12 Jan 2009 08:10:55 -0500
Subject: [Numpy-discussion] memmap from fd?
References: <gk7ehe$unj$1@ger.gmane.org>
	<3d375d730901090410h3b5f57caqe8a6b639809047a4@mail.gmail.com>
	<gk7lpr$q9j$1@ger.gmane.org>
	<3d375d730901091310v27033d5co1fbdc71098b9e672@mail.gmail.com>
	<gk97cv$egf$1@ger.gmane.org>
	<3d375d730901101741p22facd24qe6b391a0b023bd6f@mail.gmail.com>
	<gke9i6$th6$1@ger.gmane.org>
	<3d375d730901111845vf02f364g87f78dbeeea292bf@mail.gmail.com>
Message-ID: <gkffgv$ul0$1@ger.gmane.org>

Robert Kern wrote:

...
>>
>> Hmm.  frombuffer sounds nice, but python mmap doesn't expose buffer
>> interface (I just added a feature request for this).  I suppose I could
>> write my own mmap module.
> 
> Yeah, it does. Why do you think it doesn't? If it didn't work, the
> memmap class wouldn't work.
> 
My mistake.





From jh at physics.ucf.edu  Mon Jan 12 09:08:54 2009
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Mon, 12 Jan 2009 09:08:54 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <mailman.4122.1231765882.2878.numpy-discussion@scipy.org>
	(numpy-discussion-request@scipy.org)
References: <mailman.4122.1231765882.2878.numpy-discussion@scipy.org>
Message-ID: <wl8iqokipl5.fsf@glup.physics.ucf.edu>

> ... citations ... BibTeX

Please ensure that, whatever you come up with, it is *very intuitive*
for the writer to enter the citation data, or we will quickly lose
citations, and possibly writers.  For example, I would not have them
type in BibTeX entries, but rather provide boxes where they can enter
the title, author, etc. and have the system generate the BibTeX.  Not
everyone uses BibTeX, and the rules about braces, what fields are
required, etc., are not totally trivial.  Perhaps there's a web site
out there that does this already, if BibTex is to be our internal
format (which I'm not at all opposed to; I use it myself).

If a large majority of our current writers thinks BibTeX is ok, then
I'll change my mind.  We should poll them on the acceptability of any
proposed system before implementing it.

For citation keys, what's wrong with good old author-year format?
Most scientific journals use it (Abt 1985).

--jh--

Abt, H. 1985.  Harold Abt used to publish surveys of things like
  citations when he was ApJ editor in the 1980s but I'm making this
  one up just to demonstrate the format except he'd never allow an
  article title in a citation because he wanted to save ink which
  everyone hated so other journals keep the titles and we should too.
  ApJ 123:1-23.


From Eric.Lebigot at normalesup.org  Mon Jan 12 09:21:21 2009
From: Eric.Lebigot at normalesup.org (Eric LEBIGOT)
Date: Mon, 12 Jan 2009 15:21:21 +0100 (CET)
Subject: [Numpy-discussion] Fast function application on list of 2D points?
Message-ID: <alpine.OSX.2.00.0901121431090.6432@kroll.spectro.jussieu.fr>

Hello,

What is the fastest way of applying a function on a list of 2D points?  More 
specifically, I have a list of 2D points, and some do not meet some criteria 
and must be rejected.  Even more specifically, the filter only lets through 
points whose x coordinate satisfies some condition, _and_ whose y coordinates 
satisfies another condition (maybe is there room for optimization, here?).

Currently, I use

   points = numpy.apply_along_axis(filter, axis = 1, arr = points)

but this creates a bottleneck in my program (array arr may contains 1 million 
points, for instance).

Is there anything that could be faster?

Any suggestion would be much appreciated!

EOL



From pjssilva at ime.usp.br  Mon Jan 12 09:33:08 2009
From: pjssilva at ime.usp.br (Paulo J. S. Silva)
Date: Mon, 12 Jan 2009 11:33:08 -0300
Subject: [Numpy-discussion] Fast function application on list of 2D
 points?
In-Reply-To: <alpine.OSX.2.00.0901121431090.6432@kroll.spectro.jussieu.fr>
References: <alpine.OSX.2.00.0901121431090.6432@kroll.spectro.jussieu.fr>
Message-ID: <1231770788.6170.3.camel@trinity>

Why you don't create a mask to select only the points in array that
satisfies the condition on x and y coordinates. For example the code
below applies filter only to the values that have x coordinate bigger
than 0.7 and y coordinate smaller than 0.3:

    mask = numpy.logical_and(points[:,0] > 0.7, points[:,1] < 0.3)
    points = numpy.apply_along_axis(filter, axis = 1, arr = points[mask,:])

best,

Paulo

Em Seg, 2009-01-12 ?s 15:21 +0100, Eric LEBIGOT escreveu:
> Hello,
> 
> What is the fastest way of applying a function on a list of 2D points?  More 
> specifically, I have a list of 2D points, and some do not meet some criteria 
> and must be rejected.  Even more specifically, the filter only lets through 
> points whose x coordinate satisfies some condition, _and_ whose y coordinates 
> satisfies another condition (maybe is there room for optimization, here?).
> 
> Currently, I use
> 
>    points = numpy.apply_along_axis(filter, axis = 1, arr = points)
> 
> but this creates a bottleneck in my program (array arr may contains 1 million 
> points, for instance).
> 
> Is there anything that could be faster?
> 
> Any suggestion would be much appreciated!
> 
> EOL
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From pjssilva at ime.usp.br  Mon Jan 12 09:50:05 2009
From: pjssilva at ime.usp.br (Paulo J. S. Silva)
Date: Mon, 12 Jan 2009 11:50:05 -0300
Subject: [Numpy-discussion] Fast function application on list of 2D
 points?
In-Reply-To: <1231770788.6170.3.camel@trinity>
References: <alpine.OSX.2.00.0901121431090.6432@kroll.spectro.jussieu.fr>
	<1231770788.6170.3.camel@trinity>
Message-ID: <1231771805.6170.11.camel@trinity>

Eric,

You question raised my attention due to a recent post of mine related to
the same kind of problem. I was solving it without using
apply_along_axis (due to ignorance).

However I tried to use apply_along_axis to solve my problem and it
proved to be very slow. Try the following:

----------- 

import numpy as np
import time

def filter(x):
    return x.sum()

a = np.random.random((2, 1000000))

# Apply filter to all points, version 1
t = time.clock()
sums1 = np.apply_along_axis(filter, axis=0, arr=a)
print 'Elapsed time', time.clock() - t

# Apply filter to all points, version 2
t = time.clock()
sums2 = np.array([filter(p) for p in a.T])
print 'Elapsed time', time.clock() - t

print sums1 == sums2

------------

In my computer the first version takes more than 6.5 longer than the
second. However the version 2 is using list comprehensions instead of a
numpy function. I would expected it to be slower. It looks like
apply_along_axis is creating many temporary arrays.

Eric, it looks like you should try something along the second version
above and see if it is faster in your case too.

Paulo



From aisaac at american.edu  Mon Jan 12 12:41:17 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 12 Jan 2009 12:41:17 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <wl8iqokipl5.fsf@glup.physics.ucf.edu>
References: <mailman.4122.1231765882.2878.numpy-discussion@scipy.org>
	<wl8iqokipl5.fsf@glup.physics.ucf.edu>
Message-ID: <496B80BD.8050102@american.edu>

On 1/12/2009 9:08 AM jh at physics.ucf.edu apparently wrote:
> For citation keys, what's wrong with good old author-year format? 
> Most scientific journals use it (Abt 1985). 
> Abt, H. 1985.  Harold Abt used to publish surveys of things like 
>   citations when he was ApJ editor in the 1980s but I'm making this
>   one up just to demonstrate the format except he'd never allow an
>   article title in a citation because he wanted to save ink which
>   everyone hated so other journals keep the titles and we should too.
>   ApJ 123:1-23.

There are two problems.

1. The only real problem is that it is not a reST citation,
so links to the citation will not be generated, and you cannot
control the citation formatting separately.  You can still
of course use normal internal targets:

See `(Abt 1985)`_.
Some more text, and later ...

.. _(Abt 1985):

Abt, H. 1985.  Harold Abt used to publish surveys of things
like citations when he was ApJ editor in the 1980s but I'm
making this one up just to demonstrate the format except
he'd never allow an article title in a citation because he
wanted to save ink which everyone hated so other journals
keep the titles and we should too.  ApJ 123:1-23.

2. The failure to use citation keys implies that
there will be neither a citation database nor consistent
formatting.  (Say, if the docs become a book.) I do not
suppose this is *currently* a real problem.

Cheers,
Alan Isaac





From jh at physics.ucf.edu  Mon Jan 12 14:31:45 2009
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Mon, 12 Jan 2009 14:31:45 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <mailman.7.1231783202.18296.numpy-discussion@scipy.org>
	(numpy-discussion-request@scipy.org)
References: <mailman.7.1231783202.18296.numpy-discussion@scipy.org>
Message-ID: <wl83afoian2.fsf@glup.physics.ucf.edu>

Well, best to have the full functionality.  These things are intended
to be both book components and stand-alone pages.  Citation formats
are a religious war anyway.

--jh--

Date: Mon, 12 Jan 2009 12:41:17 -0500
From: Alan G Isaac <aisaac at american.edu>
Subject: Re: [Numpy-discussion] coding style: citations
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID: <496B80BD.8050102 at american.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 1/12/2009 9:08 AM jh at physics.ucf.edu apparently wrote:
> For citation keys, what's wrong with good old author-year format? 
> Most scientific journals use it (Abt 1985). 
> Abt, H. 1985.  Harold Abt used to publish surveys of things like 
>   citations when he was ApJ editor in the 1980s but I'm making this
>   one up just to demonstrate the format except he'd never allow an
>   article title in a citation because he wanted to save ink which
>   everyone hated so other journals keep the titles and we should too.
>   ApJ 123:1-23.

There are two problems.

1. The only real problem is that it is not a reST citation,
so links to the citation will not be generated, and you cannot
control the citation formatting separately.  You can still
of course use normal internal targets:

See `(Abt 1985)`_.
Some more text, and later ...

.. _(Abt 1985):

Abt, H. 1985.  Harold Abt used to publish surveys of things
like citations when he was ApJ editor in the 1980s but I'm
making this one up just to demonstrate the format except
he'd never allow an article title in a citation because he
wanted to save ink which everyone hated so other journals
keep the titles and we should too.  ApJ 123:1-23.

2. The failure to use citation keys implies that
there will be neither a citation database nor consistent
formatting.  (Say, if the docs become a book.) I do not
suppose this is *currently* a real problem.

Cheers,
Alan Isaac



From tom.denniston at alum.dartmouth.org  Mon Jan 12 14:41:24 2009
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Mon, 12 Jan 2009 13:41:24 -0600
Subject: [Numpy-discussion] Is there anyway to seed random numbers without
	global state?
Message-ID: <d6f2d3dd0901121141s22b84771maff8bfe373887a2f@mail.gmail.com>

I know how to seed and generate random numbers using:
numpy.random.seed and numpy.random.rand

The problem is the seeding of the random numbers is global which I
would think would make it non-thread safe as well as having all the
other annoyances of global state like having so set the seed and set
it back when done.

I would think the ideal would be to be able to build random number
objects where each has a seed so that each object's seed is
independent.  Does such a thing exist in numpy or scipy.  Does this
even make sense to people?  Is there something about random number
generation that makes it inherently a global thing?

--Tom


From robert.kern at gmail.com  Mon Jan 12 14:50:20 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 12 Jan 2009 13:50:20 -0600
Subject: [Numpy-discussion] Is there anyway to seed random numbers
	without global state?
In-Reply-To: <d6f2d3dd0901121141s22b84771maff8bfe373887a2f@mail.gmail.com>
References: <d6f2d3dd0901121141s22b84771maff8bfe373887a2f@mail.gmail.com>
Message-ID: <3d375d730901121150o2e3d5448tf9d22d5cc106ff27@mail.gmail.com>

On Mon, Jan 12, 2009 at 13:41, Tom Denniston
<tom.denniston at alum.dartmouth.org> wrote:
> I know how to seed and generate random numbers using:
> numpy.random.seed and numpy.random.rand
>
> The problem is the seeding of the random numbers is global which I
> would think would make it non-thread safe as well as having all the
> other annoyances of global state like having so set the seed and set
> it back when done.
>
> I would think the ideal would be to be able to build random number
> objects where each has a seed so that each object's seed is
> independent.  Does such a thing exist in numpy or scipy.  Does this
> even make sense to people?  Is there something about random number
> generation that makes it inherently a global thing?

from numpy.random import RandomState

prng = RandomState(myseed)
prng.standard_normal()
prng.uniform(low, high)
...


All of the "functions" in numpy.random are just aliases to the methods
on a global RandomState object provided for convenience. Whenever you
need to control the seed, you should explicitly instantiate a
RandomState object and pass it around.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Mon Jan 12 14:52:01 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 12 Jan 2009 20:52:01 +0100
Subject: [Numpy-discussion] Is there anyway to seed random
	numbers	without global state?
In-Reply-To: <d6f2d3dd0901121141s22b84771maff8bfe373887a2f@mail.gmail.com>
References: <d6f2d3dd0901121141s22b84771maff8bfe373887a2f@mail.gmail.com>
Message-ID: <20090112195201.GA10627@phare.normalesup.org>

On Mon, Jan 12, 2009 at 01:41:24PM -0600, Tom Denniston wrote:
> I would think the ideal would be to be able to build random number
> objects where each has a seed so that each object's seed is
> independent.

You mean something like:

In [1]: import numpy as np

In [2]: rs = np.random.RandomState(seed=3)
 
In [3]: rs.rand()
Out[3]: 0.12671989922202853

:)

Ga?l


From pav at iki.fi  Mon Jan 12 16:51:41 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 12 Jan 2009 21:51:41 +0000 (UTC)
Subject: [Numpy-discussion] formatting issues, locale and co
References: <49570E2B.2000608@ar.media.kyoto-u.ac.jp>
	<e06186140812272238v2ee8ae7kf934a5e0f55b0de7@mail.gmail.com>
	<3d375d730812272246l78ac1e45u207370fd9d0ac765@mail.gmail.com>
	<49571F6F.4070102@ar.media.kyoto-u.ac.jp>
	<e06186140812272312l21f0d321w99b910d058b902b0@mail.gmail.com>
	<5b8d13220812282038n626c44aep89756d7f0fbe7930@mail.gmail.com>
	<gjelcd$qbe$1@ger.gmane.org>
	<5b8d13220812302011i5cfb76acma1527bdcbf7a83ee@mail.gmail.com>
Message-ID: <gkge1d$sl0$1@ger.gmane.org>

Wed, 31 Dec 2008 13:11:02 +0900, David Cournapeau wrote:
[clip]
> Thank you for working on this, Pauli. The problem on windows may not be
> specific to windows: the difference really is whether the formatting is
> done by python or the C runtime. It just happens that on Linux and Mac
> OS X, the strings are the same - but it could be different on other OS.
> I have not looked into C99, whether this is standardized or not (the
> size of exponent is, but I don't know about nan and inf).

C99 appears to specify for *printf (case-insensitive)

	nan, +nan, -nan, nan(WHATEVER), -nan(WHATEVER)
	inf, infinity, +inf, +infinity, -inf, -infinity

The fromfile/fromstring code now in `fix_float_format` branch recognizes 
all of these, independently of the platform.

There are also some roundtrip tests that check that this plays along as 
expected with your new float formatting code.

> We should also change pretty print of arrays, I think - although it is a
> change and may break things. Since that's how python represents the
> numbers, I guess we will have to change at some point.

It'd be nice to make also this behave as the rest of the float 
formatting. However, the current formatting of "Inf", "-Inf", "NaN" is OK 
as far as C99 is concerned.

-- 
Pauli Virtanen



From Eric.Lebigot at normalesup.org  Tue Jan 13 04:23:20 2009
From: Eric.Lebigot at normalesup.org (Eric LEBIGOT)
Date: Tue, 13 Jan 2009 10:23:20 +0100 (CET)
Subject: [Numpy-discussion] Fast function application on list of 2D
 points?
In-Reply-To: <mailman.7.1231783202.18296.numpy-discussion@scipy.org>
References: <mailman.7.1231783202.18296.numpy-discussion@scipy.org>
Message-ID: <alpine.OSX.2.00.0901131010390.14014@kroll.spectro.jussieu.fr>

Thank you so much for the suggestion, Paulo!  Selecting 2D points in a list 
by creating an array 'mask' of booleans and then using arr[mask, :] is indeed 
really fast compared to using numpy.apply_along_axis(), in my case (simple 
"larger than" tests on individual coordinates).

I had not realized that you could do "arr[mask, :]": this works great!

EOL

PS: here are the speed tests I've done on the selection of 2D points from a 
list, with the following results:

filter0: 107.2 s
filter1: 0.3 s
filter2: 9.7 s
filter3: 0.6 s

obtained with:

#!/usr/bin/env python

import numpy

def filter0(points):
   """
   Returns only those points that match the filter.
   """
   def filter(p):
     return (p[0] > 0.5) and (p[1] < 0.5)
   return points[numpy.apply_along_axis(filter, axis = 1, arr = points)]

def filter1(points):
   """
   Returns only those points that match the filter.
   """

   mask = (points[:, 0] > 0.5) & (points[:, 1] < 0.5)
   return points[mask, :]

def filter2(points):
   """
   Returns only those points that match the filter.
   """
   return numpy.array([p for p in points if ((p[0] > 0.5) and p[1] < 0.5)])

def filter3(points):
   """
   Returns only those points that match the filter.
   """

   mask = (points[:, 0] > 0.5)
   points = points[mask, :]
   mask = points[:, 1] < 0.5
   return points[mask, :]

if __name__ == '__main__':

   import timeit


   # We generate many random points:
   NUM_PTS = 1000000
   points = numpy.random.random((NUM_PTS, 2))

   # We make sure that all the filters give the same result:
   #print "Initial points:"
   #print points
   #print "Filtered points:"
   #print filter0(points)
   #print filter1(points)
   #print filter2(points)
   #print filter3(points)


   for filter_num in range(4):
     func_name = "filter%d" % filter_num
     t = timeit.Timer("%s(points)" % func_name,
                      "from __main__ import %s, points" % func_name)
     print "%s: %.1f s" % (func_name, t.timeit(number = 3))

> Date: Mon, 12 Jan 2009 11:33:08 -0300
> From: "Paulo J. S. Silva" <pjssilva at ime.usp.br>
> Subject: Re: [Numpy-discussion] Fast function application on list of
> 	2D points?
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID: <1231770788.6170.3.camel at trinity>
> Content-Type: text/plain; charset="UTF-8"
>
> Why you don't create a mask to select only the points in array that
> satisfies the condition on x and y coordinates. For example the code
> below applies filter only to the values that have x coordinate bigger
> than 0.7 and y coordinate smaller than 0.3:
>
>    mask = numpy.logical_and(points[:,0] > 0.7, points[:,1] < 0.3)
>    points = numpy.apply_along_axis(filter, axis = 1, arr = points[mask,:])
>
> best,
>
> Paulo
>
> Em Seg, 2009-01-12 ?s 15:21 +0100, Eric LEBIGOT escreveu:
>> Hello,
>>
>> What is the fastest way of applying a function on a list of 2D points?  More
>> specifically, I have a list of 2D points, and some do not meet some criteria
>> and must be rejected.  Even more specifically, the filter only lets through
>> points whose x coordinate satisfies some condition, _and_ whose y coordinates
>> satisfies another condition (maybe is there room for optimization, here?).
>>
>> Currently, I use
>>
>>    points = numpy.apply_along_axis(filter, axis = 1, arr = points)
>>
>> but this creates a bottleneck in my program (array arr may contains 1 million
>> points, for instance).
>>
>> Is there anything that could be faster?
>>
>> Any suggestion would be much appreciated!
>>
>> EOL


From scott.sinclair.za at gmail.com  Tue Jan 13 06:48:08 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Tue, 13 Jan 2009 13:48:08 +0200
Subject: [Numpy-discussion] ndarray.resize method and reference counting
Message-ID: <6a17e9ee0901130348o4781525ck4fb88072b05d352f@mail.gmail.com>

Hi,

I'm confused by the following:

>>> import numpy as np
>>> np.__version__
'1.3.0.dev6116'

# I expect this
>>> x = np.eye(3)
>>> x.resize((5,5))
>>> x = np.eye(3)
>>> y = x
>>> x.resize((5,5))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot resize an array that has been referenced or is referencing
another array in this way.  Use the resize function

# I don't expect this
>>> x = np.eye(3)
>>> x
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])
>>> x.resize((5,5), refcheck=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot resize an array that has been referenced or is referencing
another array in this way.  Use the resize function
>>> x.resize((5,5), refcheck=False)
>>> x
array([[ 1.,  0.,  0.,  0.,  1.],
       [ 0.,  0.,  0.,  1.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.]])

Is there a reference counting bug, or am I misunderstanding something
about how Python works when I type a variable's name at the prompt?

Cheers,
Scott


From stefan at sun.ac.za  Tue Jan 13 08:24:27 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 13 Jan 2009 15:24:27 +0200
Subject: [Numpy-discussion] ndarray.resize method and reference counting
In-Reply-To: <6a17e9ee0901130348o4781525ck4fb88072b05d352f@mail.gmail.com>
References: <6a17e9ee0901130348o4781525ck4fb88072b05d352f@mail.gmail.com>
Message-ID: <9457e7c80901130524k17e6015dh83e40319957320cb@mail.gmail.com>

Hi Scott

I can't reproduce the problem below.  Would you please send a
self-contained snippet?

Note that, in Python, "_" is a special variable that always points to
the last result.  In IPython there are several others.

Cheers
St?fan

2009/1/13 Scott Sinclair <scott.sinclair.za at gmail.com>:
> # I don't expect this
>>>> x = np.eye(3)
>>>> x
> array([[ 1.,  0.,  0.],
>       [ 0.,  1.,  0.],
>       [ 0.,  0.,  1.]])
>>>> x.resize((5,5), refcheck=True)
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> ValueError: cannot resize an array that has been referenced or is referencing
> another array in this way.  Use the resize function
>>>> x.resize((5,5), refcheck=False)


From igorsyl at gmail.com  Tue Jan 13 08:37:12 2009
From: igorsyl at gmail.com (Igor Sylvester)
Date: Tue, 13 Jan 2009 07:37:12 -0600
Subject: [Numpy-discussion] PyExc_IOError
Message-ID: <b3c85b560901130537l46f934a8g62c042a8247307f2@mail.gmail.com>

Has anyone had problems with dynamically loading PyExc_IOError when
embedding a cython module?
While calling init below:

*in main.c:
*void init {
  Py_Initialize();
  initmymodule();
  do_somethind();
  Py_Finalize();
}

*in module.c:*
cdef public void do_something():
    import exceptions

    print exceptions.IOError # this prints <type 'exceptions.IOError'>, so I
don't know what to make of the error below
    import numpy  # exception is thrown here, and numpy library is in
pythonpath

Exception:
Exception exceptions.ImportError: '/usr/lib/python2.5/lib-dynload/time.so:
undefined symbol: PyExc_IOError' in 'mymodule.myfunction' ignored

I'm not yet sure which library in numpy causes the exception but this
example is easy to reproduce.

I built it against *Python 2.5.1.

compile:*
gcc -c  -I/usr/lib/python2.5/site-packages/numpy-1.0.4.0002-py2.5-linux-x86_64.egg/numpy/core/include
-I/usr/bin/python/2.5.1/include/python2.5 -fPIC -fno-omit-frame-pointer
-pthread -fexceptions -ansi -g  main.c

compile line for the other c-file is identical to main.c's

*link:*
gcc -g -pthread -shared  main.o module.o  -L/usr/bin/python/2.5.1/lib *
-lpython2.5* -ldl -lm -lpthread -lutil -lstdc++
I can import the module from python without changing any environment
variables.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090113/68b95929/attachment.html>

From nouiz at nouiz.org  Tue Jan 13 09:27:37 2009
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 13 Jan 2009 09:27:37 -0500
Subject: [Numpy-discussion] inplace matrix multiplication
In-Reply-To: <3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com>
References: <2d1d7fe70901090625k53fe9415uf177e2d80645f12c@mail.gmail.com>
	<3d375d730901091331u26a0d6dcw8dffca7349fec96@mail.gmail.com>
Message-ID: <2d1d7fe70901130627j477c3c26pf55c14783000af80@mail.gmail.com>

Thanks, and sorry for the delayed reply.

Fred

On Fri, Jan 9, 2009 at 4:31 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Fri, Jan 9, 2009 at 08:25, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> > Hi,
> >
> > I would like to know how I can make a call to the blas function gemm in
> > numpy. I need a multiply and accumulate for matrix and I don't want to
> > allocate a new matrix each time I do it.
>
> You can't in numpy. With scipy.linalg.fblas.dgemm() and the right
> arguments, you can.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090113/bd7cc1e6/attachment.html>

From aisaac at american.edu  Tue Jan 13 09:31:09 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 13 Jan 2009 09:31:09 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>
References: <496A5CC0.8050605@american.edu>	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>	<496A6D16.5050501@american.edu>
	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>
Message-ID: <496CA5AD.7090301@american.edu>

> 2009/1/12 Alan G Isaac <aisaac at american.edu>:
>> Numerical keys will clearly *not* be consistent.
>> The same key will refer to different citations
>> on different pages, and key width will not be
>> uniform.


On 1/12/2009 2:35 AM St?fan van der Walt apparently wrote:
> We automatically renumber the citations to take care of this.


But this just creates a different inconsistency:
different numbers refer to the same article, and
additionally that article citation might be formatted in
so many different ways that it is difficult to determine
that it is in fact the same article.

There really is no substitute for using real cite keys
and a central database of citations.

Alan Isaac



From stefan at sun.ac.za  Tue Jan 13 09:44:33 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 13 Jan 2009 16:44:33 +0200
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496CA5AD.7090301@american.edu>
References: <496A5CC0.8050605@american.edu>
	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>
	<496A6D16.5050501@american.edu>
	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>
	<496CA5AD.7090301@american.edu>
Message-ID: <9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>

2009/1/13 Alan G Isaac <aisaac at american.edu>:
> There really is no substitute for using real cite keys
> and a central database of citations.

How do you propose getting the citations into the docstrings?  Each
docstring needs a copy of its references, and those references may be
to specific pages.  I don't recall ReST supporting the LaTeX "[key, p.
13--40]" syntax.  So, there's some work to be done.  Come forward with
either an implementation or a solid technical proposal, and I'll
gladly take it further.

St?fan


From scott.sinclair.za at gmail.com  Tue Jan 13 09:45:36 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Tue, 13 Jan 2009 16:45:36 +0200
Subject: [Numpy-discussion] ndarray.resize method and reference counting
In-Reply-To: <9457e7c80901130524k17e6015dh83e40319957320cb@mail.gmail.com>
References: <6a17e9ee0901130348o4781525ck4fb88072b05d352f@mail.gmail.com>
	<9457e7c80901130524k17e6015dh83e40319957320cb@mail.gmail.com>
Message-ID: <6a17e9ee0901130645n41fcbc4ja11796c63d342685@mail.gmail.com>

I thought it was a self contained snippet ;-)

Here's another attempt that shows "_" is the cause of my confusion.

>>> import numpy as np
>>> x = np.eye(3)
>>> x
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])
>>> x.resize((5,5))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot resize an array that has been referenced or is referencing
another array in this way.  Use the resize function
>>> _
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])

Thanks for the help,
Scott

2009/1/13 St?fan van der Walt <stefan at sun.ac.za>:
> Hi Scott
>
> I can't reproduce the problem below.  Would you please send a
> self-contained snippet?
>
> Note that, in Python, "_" is a special variable that always points to
> the last result.  In IPython there are several others.
>
> Cheers
> St?fan
>
> 2009/1/13 Scott Sinclair <scott.sinclair.za at gmail.com>:
>> # I don't expect this
>>>>> x = np.eye(3)
>>>>> x
>> array([[ 1.,  0.,  0.],
>>       [ 0.,  1.,  0.],
>>       [ 0.,  0.,  1.]])
>>>>> x.resize((5,5), refcheck=True)
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>> ValueError: cannot resize an array that has been referenced or is referencing
>> another array in this way.  Use the resize function
>>>>> x.resize((5,5), refcheck=False)
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From aisaac at american.edu  Tue Jan 13 13:20:59 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 13 Jan 2009 13:20:59 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>
References: <496A5CC0.8050605@american.edu>	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>	<496A6D16.5050501@american.edu>	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>	<496CA5AD.7090301@american.edu>
	<9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>
Message-ID: <496CDB8B.8010603@american.edu>

> 2009/1/13 Alan G Isaac <aisaac at american.edu>:
>> There really is no substitute for using real cite keys
>> and a central database of citations.

On 1/13/2009 9:44 AM St?fan van der Walt apparently wrote:
> How do you propose getting the citations into the docstrings?  Each
> docstring needs a copy of its references, and those references may be
> to specific pages.  I don't recall ReST supporting the LaTeX "[key, p.
> 13--40]" syntax.  So, there's some work to be done.  Come forward with
> either an implementation or a solid technical proposal, and I'll
> gladly take it further.


Originally I was just pointing out a problem.
Proposing a good solution requires some discussion.
The problem has also changed because of a suggestion
that the docs should be usable for book compilation,
and I am very uncertain how that is conceived.

The key question is: should each docstring actually
contain its citation, or can their be a page of
citations?

What I can offer to help with is this: automatic
generation of citation keys, and automatic formatting
of citations.  This could be used to assist docstring
authors.  With no data base, this means that
if you write a front end to accept citation info,
we can spit out reST formatted citations plus
a rule-based cite key.  If the bib info is added to a
data base, it could be a unique rule-based cite key.

But really, all I originally wanted to do is call for
an end to the uninformative use of reST footnotes in
the place of reST citations with informative cite keys.

fwiw,
Alan Isaac

PS You can always write [last-2009-sja]_, p.9. But you
are right that reST does not allow "extra" info in the
brackets with the cite key.  More generally, it does
not provide for treating the cite key as substitutable
in generated documents.  This is a big difference from
LaTeX, but recall in reST the source is meant to be a
human readable document. (Still, I'd like to see this
functionality added.)





From timmichelsen at gmx-topmail.de  Tue Jan 13 13:55:16 2009
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Tue, 13 Jan 2009 19:55:16 +0100
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496CDB8B.8010603@american.edu>
References: <496A5CC0.8050605@american.edu>	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>	<496A6D16.5050501@american.edu>	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>	<496CA5AD.7090301@american.edu>	<9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>
	<496CDB8B.8010603@american.edu>
Message-ID: <gkio2l$ahp$1@ger.gmane.org>

Hello,
I really like the discussion here.


> Originally I was just pointing out a problem.
> Proposing a good solution requires some discussion.
> The problem has also changed because of a suggestion
> that the docs should be usable for book compilation,
> and I am very uncertain how that is conceived.
> 
> The key question is: should each docstring actually
> contain its citation, or can their be a page of
> citations?
See, many of use may have
a) a folder with all PDF documents of literatures or articles
b) a file which is used to administer the above, be it Bibtex, EndNote, etc.
c) different texts which use a) by the help of b)
d) docstrings with references

In order to unify b) and d) (=reduce doubled work) I decided for keeping 
my citations in one file (references.rst) and reference these from the 
docstrings. This works quite well.

Please have a look at:
http://bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for

And the example Sphinx project at:
#3 - 
http://bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for#comment-3397

I would appreciate very much to receive any feedback on my approach.

Hope that this helps.

Kind regards,
Timmie



From aisaac at american.edu  Tue Jan 13 14:36:34 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 13 Jan 2009 14:36:34 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <gkio2l$ahp$1@ger.gmane.org>
References: <496A5CC0.8050605@american.edu>	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>	<496A6D16.5050501@american.edu>	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>	<496CA5AD.7090301@american.edu>	<9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>	<496CDB8B.8010603@american.edu>
	<gkio2l$ahp$1@ger.gmane.org>
Message-ID: <496CED42.1020109@american.edu>

On 1/13/2009 1:55 PM Tim Michelsen apparently wrote:
> Please have a look at:
> http://bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for
> 
> And the example Sphinx project at:
> #3 - 
> http://bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for#comment-3397
> 
> I would appreciate very much to receive any feedback on my approach.


If I understand, you are using bib4txt.py to do this?
Then I can offer to help if you run into problems.

Note that bib4txt can do citation reference substitution.
I cannot tell if you are doing that separately.

You did not say specifically what the docstring problem
is, but the obvious difficulty is the following:
in a document a reference points to a citation *in that document*,
while in a docstring it is not clear whether you want that.
(I.e., should the citation text be in the docstring or
in a file of citations from all the docstrings?)

Alan Isaac



From robert.kern at gmail.com  Tue Jan 13 14:46:50 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 13 Jan 2009 13:46:50 -0600
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496CED42.1020109@american.edu>
References: <496A5CC0.8050605@american.edu>
	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>
	<496A6D16.5050501@american.edu>
	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>
	<496CA5AD.7090301@american.edu>
	<9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>
	<496CDB8B.8010603@american.edu> <gkio2l$ahp$1@ger.gmane.org>
	<496CED42.1020109@american.edu>
Message-ID: <3d375d730901131146p41af0160q51414e6042249ea1@mail.gmail.com>

On Tue, Jan 13, 2009 at 13:36, Alan G Isaac <aisaac at american.edu> wrote:
> On 1/13/2009 1:55 PM Tim Michelsen apparently wrote:
>> Please have a look at:
>> http://bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for
>>
>> And the example Sphinx project at:
>> #3 -
>> http://bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for#comment-3397
>>
>> I would appreciate very much to receive any feedback on my approach.
>
>
> If I understand, you are using bib4txt.py to do this?
> Then I can offer to help if you run into problems.
>
> Note that bib4txt can do citation reference substitution.
> I cannot tell if you are doing that separately.
>
> You did not say specifically what the docstring problem
> is, but the obvious difficulty is the following:
> in a document a reference points to a citation *in that document*,
> while in a docstring it is not clear whether you want that.
> (I.e., should the citation text be in the docstring or
> in a file of citations from all the docstrings?)

In the docstring.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From timmichelsen at gmx-topmail.de  Tue Jan 13 15:15:18 2009
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Tue, 13 Jan 2009 21:15:18 +0100
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496CED42.1020109@american.edu>
References: <496A5CC0.8050605@american.edu>	<9457e7c80901111313o4866abc3w6155bf53c89ddd5d@mail.gmail.com>	<496A6D16.5050501@american.edu>	<9457e7c80901112335h1439dd35k3b971585837db22c@mail.gmail.com>	<496CA5AD.7090301@american.edu>	<9457e7c80901130644l1cfcdab1m7698f0d5b17ce59e@mail.gmail.com>	<496CDB8B.8010603@american.edu>	<gkio2l$ahp$1@ger.gmane.org>
	<496CED42.1020109@american.edu>
Message-ID: <gkison$t7n$1@ger.gmane.org>

> If I understand, you are using bib4txt.py to do this?
Yes, I use bib4txt.py.
The extension just runs bib4txt.py over all documents to extract the 
citations. Then it creates the reference file.
The bibtex file is also converted to ReSt using your tools.

But since the bibstuff needs the citation reference to be in a sting 
object, I cannot process the docstrings in order to create the 
literature file with only the referenced citations.
I therefore just include all references as a workaround.

> Then I can offer to help if you run into problems.
The problem above is where you or Georg Brandl or anyone familiar with 
the way Sphinx processes the docstrings works.

> Note that bib4txt can do citation reference substitution.
> I cannot tell if you are doing that separately.
No, I am not doing this.
I create my reference keys in JabRef. The program is cross platform has 
an automatic bibtex key generator which works according to your defined 
pattern.

=> This is just the answer to your question on reference formatting.
The scipy/numpy community just needs to agree on a common reference 
pattern. Say: author_year or authoretal_shorttitle_year and jabref or 
any other program can take care of the rest.

> You did not say specifically what the docstring problem
> is, but the obvious difficulty is the following:
> in a document a reference points to a citation *in that document*,
> while in a docstring it is not clear whether you want that.
> (I.e., should the citation text be in the docstring or
> in a file of citations from all the docstrings?)
Please take a look at the zip file from the bugreport at bitbucket.
I am keen to see what you think. You have more X-perience and may 
suggest some improvements. It comes with a todo / readme.

... nice to see, that my efforts are of use for you.

Kind regards,
Timmie



From millman at berkeley.edu  Wed Jan 14 02:21:26 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 13 Jan 2009 23:21:26 -0800
Subject: [Numpy-discussion] remove need for site.cfg on default
Message-ID: <c7009a550901132321geac2de0o9a9e675aa9257510@mail.gmail.com>

Due to the fact that I was tired of adding site.cfg to scipy and numpy
when building on Fedora and Ubuntu systems as well as a scipy ticket
(http://scipy.org/scipy/numpy/ticket/985), I decided to try and add
default system paths to numpy.distutils.  You can find out more
details on this ticket:
  http://projects.scipy.org/scipy/scipy/ticket/817

I would like to have as many people test this as possible.  So I would
like everyone, who has had to build numpy/scipy with a site.cfg
despite having installed all the dependencies in the system default
locations, to test this.  If you would please update to the numpy
trunk and build numpy and scipy without your old site.cfg.  Regardless
of whether it works or not, I would appreciate it if you could let me
know.

Thanks,
Jarrod


From asbach at ient.rwth-aachen.de  Wed Jan 14 06:17:40 2009
From: asbach at ient.rwth-aachen.de (Mark Asbach)
Date: Wed, 14 Jan 2009 12:17:40 +0100
Subject: [Numpy-discussion] Trying to implement the array interface
Message-ID: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>

Hi there,

I'm currently extending the Python wrapper for the Open Computer  
Vision Library (opencv) with the goal to interface numerical libraries  
as seemless as possible. Unfortunately, it doesn't seem to be that  
easy ;-)

What I've done so far:

- Added an __array_interface__ property to the Python side of OpenCV  
data structures (matrices and images) that uses version 3 of the  
interface definition and supplies the keys 'version', 'shape',  
'typestr', 'data' and in some cases 'strides' when we have non- 
continuos memory layout. I think, I'm compatible to http://numpy.scipy.org/array_interface.shtml 
  .
- Added parsing the __array_interface__ of Python objects passed to  
OpenCV methods. I'm a bit unsure of how to use the C/C++ side (array  
struct) and if I can expect it to be there (for example: I don't  
provide one with OpenCV). Since I intend to keep OpenCV independent of  
numpy, calling functions from numpy.h is not an option, as far as I  
can see.

The stuff described above is in the head revision of OpenCV,  
accessible via "svn co https://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv 
".

I've tried using the following packages with OpenCV this way:
- numpy (1.0.4): everything works as expected. This is the most  
important library for OpenCV users, so this is a good sign.
- pylab/matplotlib (0.91.2): seems to use numpy / scipy-core.  
Everything okay.
- PIL (1.1.6): the array interface (Python side) doesn't adhere to the  
definition -> no 'version' key, 'data' is string, not a tuple holding  
the pointer. What to do with this?
- Numeric (24.2): I can create arrays from OpenCV datatypes and print  
them. Converting to other types however always yields 'Cannot convert  
scalar to float' or 'a float is required'. Strange :-/ Numeric.array  
instances also don't carry an __array_interface__. I can however  
convert by using numpy.arrays as intermediate step.
- Gnuplot (1.7): uses Numeric, so doesn't work as well
- pymat: didn't check. Seems to use Numeric, test results cover  
Numeric 23 and Matlab 6.5 only, so this package might be dead?
- numarray: didn't check. Is there still any relevance of this package?

Best,
Mark

-- 
Mark Asbach
Institut f?r Nachrichtentechnik, RWTH Aachen University
http://www.ient.rwth-aachen.de/cms/team/m_asbach

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4403 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090114/66ef420d/attachment.bin>

From gael.varoquaux at normalesup.org  Wed Jan 14 08:05:18 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 14 Jan 2009 14:05:18 +0100
Subject: [Numpy-discussion] Trying to implement the array interface
In-Reply-To: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
References: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
Message-ID: <20090114130518.GA21830@phare.normalesup.org>

On Wed, Jan 14, 2009 at 12:17:40PM +0100, Mark Asbach wrote:
> - Added an __array_interface__ property to the Python side of OpenCV data 
> structures (matrices and images) [...]

> - Added parsing the __array_interface__ of Python objects passed to OpenCV 
> methods. [...]

> I've tried using the following packages with OpenCV this way:
> - numpy (1.0.4): everything works as expected. This is the most important 
> library for OpenCV users, so this is a good sign. [...]

_Fantastic_! I read from this e-mail that you are having a few
difficulties moving forward, but what you have already achieved is
really fantastic and will definitely be very useful to many people.

Sorry, I can't help you with the problems you are having, but I wanted to
thank you for this effort.

Ga?l


From matthieu.brucher at gmail.com  Wed Jan 14 08:37:55 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 14 Jan 2009 14:37:55 +0100
Subject: [Numpy-discussion] Trying to implement the array interface
In-Reply-To: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
References: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
Message-ID: <e76aa17f0901140537m7424dd86h80ce4cc2ed4c4f4a@mail.gmail.com>

2009/1/14 Mark Asbach <asbach at ient.rwth-aachen.de>:
> Hi there,
>
> I'm currently extending the Python wrapper for the Open Computer Vision
> Library (opencv) with the goal to interface numerical libraries as seemless
> as possible. Unfortunately, it doesn't seem to be that easy ;-)
>
> What I've done so far:
>
> - Added an __array_interface__ property to the Python side of OpenCV data
> structures (matrices and images) that uses version 3 of the interface
> definition and supplies the keys 'version', 'shape', 'typestr', 'data' and
> in some cases 'strides' when we have non-continuos memory layout. I think,
> I'm compatible to http://numpy.scipy.org/array_interface.shtml .

Great! How did you do it? I've written something like this before, but
it was always very complex :|

> - Added parsing the __array_interface__ of Python objects passed to OpenCV
> methods. I'm a bit unsure of how to use the C/C++ side (array struct) and if
> I can expect it to be there (for example: I don't provide one with OpenCV).
> Since I intend to keep OpenCV independent of numpy, calling functions from
> numpy.h is not an option, as far as I can see.

Perhaps you can tell SWIG to extrat those informations from the numpy
array? A more complex numpy.i would probably be enough.

> - PIL (1.1.6): the array interface (Python side) doesn't adhere to the
> definition -> no 'version' key, 'data' is string, not a tuple holding the
> pointer. What to do with this?

Perhaps you can force it to pass numpy.asarray(image)?

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From oliphant at enthought.com  Wed Jan 14 09:20:03 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 14 Jan 2009 08:20:03 -0600
Subject: [Numpy-discussion] Trying to implement the array interface
In-Reply-To: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
References: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
Message-ID: <496DF493.6070502@enthought.com>

Mark Asbach wrote:
> Hi there,
>
> I'm currently extending the Python wrapper for the Open Computer 
> Vision Library (opencv) with the goal to interface numerical libraries 
> as seemless as possible. Unfortunately, it doesn't seem to be that 
> easy ;-)
>
> What I've done so far:
>
> - Added an __array_interface__ property to the Python side of OpenCV 
> data structures (matrices and images) that uses version 3 of the 
> interface definition and supplies the keys 'version', 'shape', 
> 'typestr', 'data' and in some cases 'strides' when we have 
> non-continuos memory layout. I think, I'm compatible to 
> http://numpy.scipy.org/array_interface.shtml .
Great.   This is a good first step.
> - Added parsing the __array_interface__ of Python objects passed to 
> OpenCV methods. I'm a bit unsure of how to use the C/C++ side (array 
> struct) and if I can expect it to be there (for example: I don't 
> provide one with OpenCV). Since I intend to keep OpenCV independent of 
> numpy, calling functions from numpy.h is not an option, as far as I 
> can see.
You can't expect the __array_struct__ property to be there, but if it 
is, it allows you to get all the information you need with one attribute 
lookup (albeit in C) instead of many.
>
> The stuff described above is in the head revision of OpenCV, 
> accessible via "svn co 
> https://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv". 
>
>
> I've tried using the following packages with OpenCV this way:
> - numpy (1.0.4): everything works as expected. This is the most 
> important library for OpenCV users, so this is a good sign.
> - pylab/matplotlib (0.91.2): seems to use numpy / scipy-core. 
> Everything okay.
> - PIL (1.1.6): the array interface (Python side) doesn't adhere to the 
> definition -> no 'version' key, 'data' is string, not a tuple holding 
> the pointer. What to do with this?
That is probably true.   I've worked a bit with PIL to get things to 
work, but haven't followed the project lately to see where it is at.   
One difficulty is that the PIL memory layout can be quite different from 
a NumPy array, and so that would be why the "data" is a string.   The 
best source for implementing consuming of the interface is to look in 
the NumPy source code and look for where it grabs the 
__array_interface__ and/or __array_struct__ attribute and makes use of 
the data found there.
> - Numeric (24.2): I can create arrays from OpenCV datatypes and print 
> them. Converting to other types however always yields 'Cannot convert 
> scalar to float' or 'a float is required'. Strange :-/ Numeric.array 
> instances also don't carry an __array_interface__. I can however 
> convert by using numpy.arrays as intermediate step.
I believe the __array_struct__ property was used instead.     You can 
implement either __array_struct__ or __array_interface__ or both as an 
exporter.   Thus, a consumer that wants to see everything has to consume 
both.   I'm not sure I understand the error you are getting exactly.  
> - Gnuplot (1.7): uses Numeric, so doesn't work as well
> - pymat: didn't check. Seems to use Numeric, test results cover 
> Numeric 23 and Matlab 6.5 only, so this package might be dead?
I don't remember pymat very well.
> - numarray: didn't check. Is there still any relevance of this package? 
A few tools still use it, but it is deprecated (as is Numeric of 
course).   

I'm glad to see you using the __array_interface__ because it will allow 
you to share memory with numXXX arrays.  I'm sure you are also aware of 
the new buffer protocol in Python 2.6 and Python 3.0.   This is the 
approach to take for the future.  NumPy, of course, will support the 
__array_interface__ and __array_struct__ properties for some time.   I'm 
hoping that a new release of NumPy will also support the new Python 
buffer interface.   But, as of yet, it does not.

Best regards,

-Travis



From millman at berkeley.edu  Wed Jan 14 09:34:39 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 14 Jan 2009 06:34:39 -0800
Subject: [Numpy-discussion] Trying to implement the array interface
In-Reply-To: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
References: <E3163E7F-049F-40EE-8EC4-3A3FF4107ECC@ient.rwth-aachen.de>
Message-ID: <c7009a550901140634p5c8fbde5o4e5b90d336c4193e@mail.gmail.com>

On Wed, Jan 14, 2009 at 3:17 AM, Mark Asbach <asbach at ient.rwth-aachen.de> wrote:
> I'm currently extending the Python wrapper for the Open Computer Vision
> Library (opencv) with the goal to interface numerical libraries as seemless
> as possible. Unfortunately, it doesn't seem to be that easy ;-)

This is really great.  Thanks for working on this.

> - pymat: didn't check. Seems to use Numeric, test results cover Numeric 23
> and Matlab 6.5 only, so this package might be dead?

It is pretty much dead.  Take a look at the mlabwrap scikit:
    http://scikits.appspot.com/mlabwrap
mlabwrap hasn't been updated since 2007, but it works pretty well,
supports numpy, and is much better than pymat.

Jarrod


From jackchungchiehyu at googlemail.com  Wed Jan 14 14:57:50 2009
From: jackchungchiehyu at googlemail.com (Jack Yu)
Date: Wed, 14 Jan 2009 19:57:50 +0000 (UTC)
Subject: [Numpy-discussion] Complete LAPACK needed (Frank Lagor)
References: <9fddf64a0811011030t92d10f2q8419b3348821bd10@mail.gmail.com>
Message-ID: <loom.20090114T193230-28@post.gmane.org>

Frank Lagor <dfranci <at> seas.upenn.edu> writes:

> 
> Thanks so much for your help, David.  I'm sorry I did not receive your posts
previously -- I have the digest mode on and there is a bit of a delay.  I'll try
to change my options next time I post a request. 
> Thanks so much again,Frank
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion <at> scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


Hello,
I'm trying to run a python script in Condor(http://www.cs.wisc.edu/condor/) on a
cluster at uni. This script runs fine when I run it normally, just by typing
'python', followed by the name of the script.  However, after writing a condor
submission file for it, and submitting it as a condor job, it fails at the
importing of numpy, giving:
========================================
Traceback (most recent call last):
  File "/home/spxjcy/mycondor_stuff/CountApples/xCountApples.py", line 9, in <mo
dule>
    import numpy
  File "/opt/bin/python/lib/python2.5/site-packages/numpy/__init__.py", line 103
, in <module>
    import linalg
  File "/opt/bin/python/lib/python2.5/site-packages/numpy/linalg/__init__.py", l
ine 4, in <module>
    from linalg import *
  File "/opt/bin/python/lib/python2.5/site-packages/numpy/linalg/linalg.py", lin
e 29, in <module>
    from numpy.linalg import lapack_lite
ImportError: liblapack.so.3: cannot open shared object file: No such file or dir
ectory
========================================

After reading your thread, I realised that your problem was not the same, but
the closest I could find.  I tried 'numpy.test()', nothing fails.  On 'ldd
lapack_lite.so' I get:

========================================
        liblapack.so.3 => /usr/lib64/liblapack.so.3 (0x00002aaaaabd6000)
        libblas.so.3 => /usr/lib64/libblas.so.3 (0x00002aaaab1c2000)
        libg2c.so.0 => /usr/lib64/libg2c.so.0 (0x00002aaaab314000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaab435000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab5bb000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaab6c7000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
========================================

I'm not entirely sure what this means, but it looks like there is a file called
liblapack.so.3

Another thing I have observed is that other modules which came with python like
'os', 're',etc. all import fine in condor, so I'm guessing that this is to do
with numpy on some part.  

Please, if you have any ideas about solving this, do say.  

thanks,

Jack  



From dsdale24 at gmail.com  Wed Jan 14 20:29:34 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 14 Jan 2009 20:29:34 -0500
Subject: [Numpy-discussion] any news concerning numpy for python-2.6 on
	windows?
Message-ID: <a08e5f80901141729k417463d2h7c431f1b53c24a6b@mail.gmail.com>

Hello,

I've been developing a numpy based package that I'd like to share with this
list, but I need to test it on windows first. Is there any news concerning
numpy installers for python-2.6 on windows, and is there any chance of
installing on Vista?

Thank you,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090114/7890eda4/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Jan 14 20:35:39 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 15 Jan 2009 10:35:39 +0900
Subject: [Numpy-discussion] any news concerning numpy for python-2.6 on
 windows?
In-Reply-To: <a08e5f80901141729k417463d2h7c431f1b53c24a6b@mail.gmail.com>
References: <a08e5f80901141729k417463d2h7c431f1b53c24a6b@mail.gmail.com>
Message-ID: <496E92EB.2000800@ar.media.kyoto-u.ac.jp>

Darren Dale wrote:
> Hello,
>
> I've been developing a numpy based package that I'd like to share with
> this list, but I need to test it on windows first. Is there any news
> concerning numpy installers for python-2.6 on windows, and is there
> any chance of installing on Vista?

There will be 2.6 installers for numpy 1.3. I am not aware of any vista
problems as long as python itself and numpy are installed for all users
- I would like to know about them otherwise,

David


From slaunger at gmail.com  Thu Jan 15 05:48:23 2009
From: slaunger at gmail.com (Kim Hansen)
Date: Thu, 15 Jan 2009 11:48:23 +0100
Subject: [Numpy-discussion] How to handle fields less than one byte in a
	recarray
Message-ID: <f2e7a4240901150248w127ff7ebi25e632392a6d3791@mail.gmail.com>

Hi Numpy forum

Let me start out with a generic example:

In [3]: test_byte_str = "".join(["\x12\x03", "\x23\x05", "\x35\x08"])

In [4]: desc = dtype({'names' : ["HIGHlow", "HIGH + low"], 'formats': [uint8, ui
nt8]})

In [5]: r = rec.fromstring(test_byte_str, dtype=desc)

In [6]: r[0]
Out[6]: (18, 3)

In [7]: r['HIGHlow']
Out[7]: array([18, 35, 53], dtype=uint8)

In [8]: high = r['HIGHlow'] >> 4

In [9]: high
Out[9]: array([1, 2, 3], dtype=uint8)

In [10]: low = r['HIGHlow'] & 0x0F

In [11]: low
Out[11]: array([2, 3, 5], dtype=uint8)

In [12]: high_plus_low = high + low

In [13]: high_plus_low
Out[13]: array([3, 5, 8], dtype=uint8)

In [14]: r["HIGH + low"]
Out[14]: array([3, 5, 8], dtype=uint8)

So I have here an array, where the first byte actually represents two
fields, HIGH and low, where HIGH are the four most significant bits
and low are the four least significant bits.

As you can see in the example I can figure out how to get an array of
HIGH and low by postprocessing the recordarray, but I would really
like to encapsulate this logic somehow inside the recordarray (or in
an instance of derived class), such that I could access the arrays
directly as

r["HIGH"] and r["low"]

or perhaps even better wrap it all up inside a class RWrapper, such that

r = RWrapper(test_byte_str)
r.low

would work

but not

r.HIGHlow

How do I implment that in a Pythonic way?

My real application is considerebaly more complicated, but I think
this describes the main problem.


From gael.varoquaux at normalesup.org  Thu Jan 15 05:54:55 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 15 Jan 2009 11:54:55 +0100
Subject: [Numpy-discussion] Build errors
Message-ID: <20090115105455.GB32576@phare.normalesup.org>

OK, here we go for the stupid questions showing that I really don't
understaind building well:

I am building numpy on a Mandriva x86_64. I built an optimized ATLAS, and
added the relevant lines to the site.cfg so that numpy does find the
libraries. But now I get the following error at build:

/usr/bin/ld: /volatile/varoquau/usr/lib//libptcblas.a(cblas_dptgemm.o):
relocation R_X86_64_32 against `a local symbol' can not be used when
making a shared object; recompile with -fPIC
/volatile/varoquau/usr/lib//libptcblas.a: could not read symbols: Bad
value
collect2: ld returned 1 exit status

I must confess I really have no clue what this means, and how I solve
this.

Cheers,

Ga?l


From michael.abshoff at googlemail.com  Thu Jan 15 06:11:22 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 15 Jan 2009 03:11:22 -0800
Subject: [Numpy-discussion] Build errors
In-Reply-To: <20090115105455.GB32576@phare.normalesup.org>
References: <20090115105455.GB32576@phare.normalesup.org>
Message-ID: <496F19DA.5080204@gmail.com>

Gael Varoquaux wrote:

Hi Gael,

> OK, here we go for the stupid questions showing that I really don't
> understaind building well:
> 
> I am building numpy on a Mandriva x86_64. I built an optimized ATLAS, and
> added the relevant lines to the site.cfg so that numpy does find the
> libraries. But now I get the following error at build:
> 
> /usr/bin/ld: /volatile/varoquau/usr/lib//libptcblas.a(cblas_dptgemm.o):
> relocation R_X86_64_32 against `a local symbol' can not be used when
> making a shared object; recompile with -fPIC
> /volatile/varoquau/usr/lib//libptcblas.a: could not read symbols: Bad
> value
> collect2: ld returned 1 exit status
> 
> I must confess I really have no clue what this means, and how I solve
> this.

You need to build dynamic version of ATLAS or alternatively make ATLAS 
use -fPIC during compilation when building static libs. Note that AFAIK 
ATLAS 3.8.2's make install does not copy over the dynamic libs, but you 
should be easily be able to copy them over manually.

The patch I am using is

--- Make.top.orig	2009-01-01 19:20:21.000000000 -0800
+++ Make.top	2008-03-20 02:26:35.000000000 -0700
@@ -298,5 +298,11 @@
  	- chmod 0644 $(INSTdir)/libf77blas.a
  	- cp $(LIBdir)/libptcblas.a $(INSTdir)/.
  	- cp $(LIBdir)/libptf77blas.a $(INSTdir)/.
+	- cp $(LIBdir)/libatlas.so $(INSTdir)/.
+	- cp $(LIBdir)/libcblas.so $(INSTdir)/.
+	- cp $(LIBdir)/libf77blas.so $(INSTdir)/.
+	- cp $(LIBdir)/liblapack.so $(INSTdir)/.
+	- chmod 0644 $(INSTdir)/libatlas.so $(INSTdir)/liblapack.so \
+		     $(INSTdir)/libcblas.so $(INSTdir)/libcblas.so
  	- chmod 0644 $(INSTdir)/libptcblas.a $(INSTdir)/libptf77blas.a


but you would need to add the appropriate lines for the multi threaded 
libs, too. The install issue is fixed in ATLAS 3.9.x AFAIK.

> Cheers,
> 
> Ga?l

Cheers,

Michael
Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From dsdale24 at gmail.com  Thu Jan 15 07:46:37 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Thu, 15 Jan 2009 07:46:37 -0500
Subject: [Numpy-discussion] any news concerning numpy for python-2.6 on
	windows?
In-Reply-To: <496E92EB.2000800@ar.media.kyoto-u.ac.jp>
References: <a08e5f80901141729k417463d2h7c431f1b53c24a6b@mail.gmail.com>
	<496E92EB.2000800@ar.media.kyoto-u.ac.jp>
Message-ID: <a08e5f80901150446p2d7c5b18labd02e034a533a78@mail.gmail.com>

On Wed, Jan 14, 2009 at 8:35 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Darren Dale wrote:
> > Hello,
> >
> > I've been developing a numpy based package that I'd like to share with
> > this list, but I need to test it on windows first. Is there any news
> > concerning numpy installers for python-2.6 on windows, and is there
> > any chance of installing on Vista?
>
> There will be 2.6 installers for numpy 1.3. I am not aware of any vista
> problems as long as python itself and numpy are installed for all users
> - I would like to know about them otherwise,
>

Thanks David. Any ideas about the release date?
http://projects.scipy.org/scipy/numpy/roadmap is not up to date.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090115/961f4afe/attachment.html>

From gael.varoquaux at normalesup.org  Thu Jan 15 07:55:14 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 15 Jan 2009 13:55:14 +0100
Subject: [Numpy-discussion] Build errors
In-Reply-To: <496F19DA.5080204@gmail.com>
References: <20090115105455.GB32576@phare.normalesup.org>
	<496F19DA.5080204@gmail.com>
Message-ID: <20090115125514.GA26690@phare.normalesup.org>

On Thu, Jan 15, 2009 at 03:11:22AM -0800, Michael Abshoff wrote:
> You need to build dynamic version of ATLAS or alternatively make ATLAS 
> use -fPIC during compilation when building static libs. Note that AFAIK 
> ATLAS 3.8.2's make install does not copy over the dynamic libs, but you 
> should be easily be able to copy them over manually.

Of course. I was beeing brain dead. I did this last time, and this time I
forgot. I had also forgetten to optimise lapack.... Doing things too
quickly.

Thanks,

Ga?l


From numpy-discussion at maubp.freeserve.co.uk  Thu Jan 15 07:51:50 2009
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Thu, 15 Jan 2009 12:51:50 +0000
Subject: [Numpy-discussion] any news concerning numpy for python-2.6 on
	windows?
In-Reply-To: <496E92EB.2000800@ar.media.kyoto-u.ac.jp>
References: <a08e5f80901141729k417463d2h7c431f1b53c24a6b@mail.gmail.com>
	<496E92EB.2000800@ar.media.kyoto-u.ac.jp>
Message-ID: <320fb6e00901150451t122871dfv1e3d6aad486bba11@mail.gmail.com>

On Thu, Jan 15, 2009 at 1:35 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
>
> There will be 2.6 installers for numpy 1.3. I am not aware of any vista
> problems as long as python itself and numpy are installed for all users
> - I would like to know about them otherwise,
>
> David

Are there any plans to back-port the required fixes to the numpy 1.2
branch in order to provide numpy 1.2 on Windows?  If numpy 1.3 is due
out fairly shortly, this isn't so pressing I suppose.

However, having numpy 1.2 available would be helpful for Windows
users, especially if it allows them a more gradual transition from
numpy 1.1 to numpy 1.3 as I'm sure there will be a few
deprecations/API changes to deal with.

Thanks,

Peter


From cournape at gmail.com  Thu Jan 15 08:36:13 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 15 Jan 2009 22:36:13 +0900
Subject: [Numpy-discussion] any news concerning numpy for python-2.6 on
	windows?
In-Reply-To: <320fb6e00901150451t122871dfv1e3d6aad486bba11@mail.gmail.com>
References: <a08e5f80901141729k417463d2h7c431f1b53c24a6b@mail.gmail.com>
	<496E92EB.2000800@ar.media.kyoto-u.ac.jp>
	<320fb6e00901150451t122871dfv1e3d6aad486bba11@mail.gmail.com>
Message-ID: <5b8d13220901150536o47da2e7ve6fcdb7b61f36a3e@mail.gmail.com>

On Thu, Jan 15, 2009 at 9:51 PM, Peter
<numpy-discussion at maubp.freeserve.co.uk> wrote:
> On Thu, Jan 15, 2009 at 1:35 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>>
>> There will be 2.6 installers for numpy 1.3. I am not aware of any vista
>> problems as long as python itself and numpy are installed for all users
>> - I would like to know about them otherwise,
>>
>> David
>
> Are there any plans to back-port the required fixes to the numpy 1.2
> branch in order to provide numpy 1.2 on Windows?  If numpy 1.3 is due
> out fairly shortly, this isn't so pressing I suppose.

No, the changes required are too significant, unfortunately. In
principle, it is of course possible, but that would take too much
time.

>
> However, having numpy 1.2 available would be helpful for Windows
> users, especially if it allows them a more gradual transition from
> numpy 1.1 to numpy 1.3 as I'm sure there will be a few
> deprecations/API changes to deal with.

Numpy 1.2 is already availableon windows  - only not for python2.6,

David


From ndbecker2 at gmail.com  Thu Jan 15 09:27:04 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 15 Jan 2009 09:27:04 -0500
Subject: [Numpy-discussion] limit function
Message-ID: <gknh3p$hc4$1@ger.gmane.org>

Is there a function to apply a limit to an array?  I want to (efficiently) do:

y = x if x < limit, otherwise limit




From cimrman3 at ntc.zcu.cz  Thu Jan 15 09:31:01 2009
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Thu, 15 Jan 2009 15:31:01 +0100
Subject: [Numpy-discussion] limit function
In-Reply-To: <gknh3p$hc4$1@ger.gmane.org>
References: <gknh3p$hc4$1@ger.gmane.org>
Message-ID: <496F48A5.6050407@ntc.zcu.cz>

Neal Becker wrote:
> Is there a function to apply a limit to an array?  I want to (efficiently) do:
> 
> y = x if x < limit, otherwise limit

What about np.clip?

r.



From cournape at gmail.com  Thu Jan 15 09:32:32 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 15 Jan 2009 23:32:32 +0900
Subject: [Numpy-discussion] limit function
In-Reply-To: <gknh3p$hc4$1@ger.gmane.org>
References: <gknh3p$hc4$1@ger.gmane.org>
Message-ID: <5b8d13220901150632p67ea871by582c280f6107055c@mail.gmail.com>

On Thu, Jan 15, 2009 at 11:27 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> Is there a function to apply a limit to an array?  I want to (efficiently) do:
>
> y = x if x < limit, otherwise limit

Would clip do ? For native types, it should be relatively fast, in
particular if you use the argout argument,

David


From dsdale24 at gmail.com  Thu Jan 15 10:18:07 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Thu, 15 Jan 2009 10:18:07 -0500
Subject: [Numpy-discussion] remove need for site.cfg on default
In-Reply-To: <c7009a550901132321geac2de0o9a9e675aa9257510@mail.gmail.com>
References: <c7009a550901132321geac2de0o9a9e675aa9257510@mail.gmail.com>
Message-ID: <a08e5f80901150718ke9d0b38l80c7e2291be56867@mail.gmail.com>

Hi Jarrod,

On Wed, Jan 14, 2009 at 2:21 AM, Jarrod Millman <millman at berkeley.edu>wrote:

> Due to the fact that I was tired of adding site.cfg to scipy and numpy
> when building on Fedora and Ubuntu systems as well as a scipy ticket
> (http://scipy.org/scipy/numpy/ticket/985), I decided to try and add
> default system paths to numpy.distutils.  You can find out more
> details on this ticket:
>  http://projects.scipy.org/scipy/scipy/ticket/817
>
> I would like to have as many people test this as possible.  So I would
> like everyone, who has had to build numpy/scipy with a site.cfg
> despite having installed all the dependencies in the system default
> locations, to test this.  If you would please update to the numpy
> trunk and build numpy and scipy without your old site.cfg.  Regardless
> of whether it works or not, I would appreciate it if you could let me
> know.
>

I tested on Gentoo, which does some non-standard stuff in an attempt to be
able to switch between implementations. The libraries in /usr/lib are
symlinks to the implementations in /usr/lib/blas/, and you can have various
implementations like atlas, reference, or threaded-atlas, so I can run a
program called eselect that lets me switch between the implementations. For
some reason, gentoo renames the f77blas libs to simply blas. As a result, I
need to make a minor change numpy/distutils/system_info.py:

$ svn diff
Index: numpy/distutils/system_info.py
===================================================================
--- numpy/distutils/system_info.py      (revision 6323)
+++ numpy/distutils/system_info.py      (working copy)
@@ -90,7 +90,7 @@
 [atlas]
 library_dirs = /usr/lib/3dnow:/usr/lib/3dnow/atlas
 # for overriding the names of the atlas libraries
-atlas_libs = lapack, f77blas, cblas, atlas
+atlas_libs = lapack, blas, cblas, atlas

 [x11]
 library_dirs = /usr/X11R6/lib
@@ -893,7 +893,7 @@
 class atlas_info(system_info):
     section = 'atlas'
     dir_env_var = 'ATLAS'
-    _lib_names = ['f77blas','cblas']
+    _lib_names = ['blas','cblas']
     if sys.platform[:7]=='freebsd':
         _lib_atlas = ['atlas_r']
         _lib_lapack = ['alapack_r']
@@ -1000,7 +1000,7 @@
         self.set_info(**info)

 class atlas_blas_info(atlas_info):
-    _lib_names = ['f77blas','cblas']
+    _lib_names = ['blas','cblas']

     def calc_info(self):
         lib_dirs = self.get_lib_dirs()


On my system , the symlinks point to the threaded atlas implementation. I
ran the full tests and all tests passed except the 9 known failures.

In case you are interested, here is the layout on gentoo

$ slocate liblapack
/usr/lib64/liblapack.so.0
/usr/lib64/liblapack.a
/usr/lib64/lapack/atlas/liblapack.so.0
/usr/lib64/lapack/atlas/liblapack.a
/usr/lib64/lapack/atlas/liblapack.la
/usr/lib64/lapack/atlas/liblapack.so
/usr/lib64/lapack/atlas/liblapack.so.0.0.0
/usr/lib64/lapack/reference/liblapack.so.0
/usr/lib64/lapack/reference/liblapack.a
/usr/lib64/lapack/reference/liblapack.la
/usr/lib64/lapack/reference/liblapack.so
/usr/lib64/lapack/reference/liblapack.so.0.0.0
/usr/lib64/liblapack.so

$ slocate lib*blas:
/usr/lib64/blas
/usr/lib64/blas/gsl
/usr/lib64/blas/gsl/cblas.pc
/usr/lib64/blas/atlas
/usr/lib64/blas/atlas/libcblas.so.0.0.0
/usr/lib64/blas/atlas/libcblas.la
/usr/lib64/blas/atlas/libcblas.so
/usr/lib64/blas/atlas/libblas.so.0
/usr/lib64/blas/atlas/libcblas.so.0
/usr/lib64/blas/atlas/libblas.la
/usr/lib64/blas/atlas/libblas.so
/usr/lib64/blas/atlas/cblas.pc
/usr/lib64/blas/atlas/libcblas.a
/usr/lib64/blas/atlas/libblas.so.0.0.0
/usr/lib64/blas/atlas/blas.pc
/usr/lib64/blas/atlas/libblas.a
/usr/lib64/blas/reference
/usr/lib64/blas/reference/libblas.so.0
/usr/lib64/blas/reference/libblas.la
/usr/lib64/blas/reference/libblas.so
/usr/lib64/blas/reference/libblas.so.0.0.0
/usr/lib64/blas/reference/blas.pc
/usr/lib64/blas/reference/libblas.a
/usr/lib64/blas/threaded-atlas
/usr/lib64/blas/threaded-atlas/libcblas.so.0.0.0
/usr/lib64/blas/threaded-atlas/libcblas.la
/usr/lib64/blas/threaded-atlas/libcblas.so
/usr/lib64/blas/threaded-atlas/libblas.so.0
/usr/lib64/blas/threaded-atlas/libcblas.so.0
/usr/lib64/blas/threaded-atlas/libblas.la
/usr/lib64/blas/threaded-atlas/libblas.so
/usr/lib64/blas/threaded-atlas/cblas.pc
/usr/lib64/blas/threaded-atlas/libcblas.a
/usr/lib64/blas/threaded-atlas/libblas.so.0.0.0
/usr/lib64/blas/threaded-atlas/blas.pc
/usr/lib64/blas/threaded-atlas/libblas.a
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux/cblas.ph
/usr/lib64/libcblas.so
/usr/lib64/pkgconfig/cblas.pc
/usr/lib64/pkgconfig/blas.pc
/usr/lib64/libgslcblas.a
/usr/lib64/libblas.so.0
/usr/lib64/libcblas.so.0
/usr/lib64/libblas.so
/usr/lib64/libcblas.a
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090115/1a279554/attachment.html>

From sturla at molden.no  Thu Jan 15 11:17:49 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 15 Jan 2009 17:17:49 +0100
Subject: [Numpy-discussion] error handling with f2py?
Message-ID: <496F61AD.8080902@molden.no>

Is it possible to make f2py raise an exception if a fortran routine 
signals an error?

If I e.g. have

     subroutine foobar(a, ierr)

Can I get an exception automatically raised if ierr != 0?


Sturla Molden



From gwg at emss.co.za  Thu Jan 15 14:45:46 2009
From: gwg at emss.co.za (George Goussard)
Date: Thu, 15 Jan 2009 19:45:46 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?Singular_Matrix_problem_with_Matplit?=
	=?utf-8?q?lib_in=09Numpy_=28Windows_-_AMD64=29?=
References: <15B34CD0955E484689D667626E6456D5011C8E787E@london.emss.co.za>	<5b8d13220812080943g69d4c670jabd6aef66d336e29@mail.gmail.com>
	<loom.20081217T094146-465@post.gmane.org>
	<4948D139.4020703@gmail.com>
	<loom.20081217T182123-184@post.gmane.org>
Message-ID: <loom.20090115T192826-15@post.gmane.org>

Hello.

I was suppose to give more information, but after about two months(I am a
Python newbie, newbie to Matplotlib and newbie to Numpy plus a vacation
in-between).....I finally tracked down the problem. However ....... I see in
Matplotlib 0.98.5.2 the correction was made by somebody else. (Cry, sob, sob
sob). The problem was in the file MPL_isnan.h on line 26.

Consider the issue closed.

Thanks.

George.




From bolme1234 at comcast.net  Thu Jan 15 17:55:38 2009
From: bolme1234 at comcast.net (David Bolme)
Date: Thu, 15 Jan 2009 15:55:38 -0700
Subject: [Numpy-discussion] Help with interpolating missing values from a 3D
	scanner
In-Reply-To: <gknb08$pnq$1@ger.gmane.org>
References: <gknb08$pnq$1@ger.gmane.org>
Message-ID: <EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>



I am working on a face recognition using 3D data from a special 3D  
imaging system.  For those interested the data comes from the FRGC  
2004 dataset.  The problem I am having is that for some pixels the  
scanner fails to capture depth information.  The result is that the  
image has missing values.  There are small regions on the face such as  
eyebrows and eyes that are missing the depth information.  I would  
like to fill in these region by interpolating from nearby pixels but I  
am not sure of the best way to do that.

I currently have two arrays:

* floating point array with depth information (missing data is encoded  
as a large negative number -999999.0)
* boolean array that is a missing data mask

I have some ideas on how to solve this problem but I am hoping that  
someone on this list may have more experience with this type of  
missing data problem.

* Are there any utilities in scipy/numpy designed for this type of  
missing data problem?
* If not does any one have suggestions on how I should proceed?





From robert.kern at gmail.com  Thu Jan 15 19:14:47 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Jan 2009 18:14:47 -0600
Subject: [Numpy-discussion] Help with interpolating missing values from
	a 3D scanner
In-Reply-To: <EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
References: <gknb08$pnq$1@ger.gmane.org>
	<EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
Message-ID: <3d375d730901151614h2c3b8478y9900bba97dbd8381@mail.gmail.com>

On Thu, Jan 15, 2009 at 16:55, David Bolme <bolme1234 at comcast.net> wrote:
>
> I am working on a face recognition using 3D data from a special 3D
> imaging system.  For those interested the data comes from the FRGC
> 2004 dataset.  The problem I am having is that for some pixels the
> scanner fails to capture depth information.  The result is that the
> image has missing values.  There are small regions on the face such as
> eyebrows and eyes that are missing the depth information.  I would
> like to fill in these region by interpolating from nearby pixels but I
> am not sure of the best way to do that.
>
> I currently have two arrays:
>
> * floating point array with depth information (missing data is encoded
> as a large negative number -999999.0)
> * boolean array that is a missing data mask
>
> I have some ideas on how to solve this problem but I am hoping that
> someone on this list may have more experience with this type of
> missing data problem.
>
> * Are there any utilities in scipy/numpy designed for this type of
> missing data problem?

You could toss it into the natural neighbor interpolator in
scikits.delaunay. It's designed for interpolating scattered (X,Y)
points onto a grid, but it works fine for interpolating a regular grid
with missing values, too.

Similarly, scipy.interpolate.Rbf should work, too.

> * If not does any one have suggestions on how I should proceed?

Another approach (that you would have to code yourself) is to take a
Gaussian smoothing kernel of an appropriate size, center it over each
missing pixel, then average the known pixels under the kernel using
the kernel as a weighting factor. Place that average value into the
missing pixel. This is actually fairly similar to the Rbf method
above, but will probably be more efficient since you know that the
points are all gridded.

If you have sizable contiguous regions of missing pixels like the
eyes, you may want to iterate that process, only including the last
iteration's values for the missing pixels into the average, too.
Iterate until your deltas between iterations is within a desirable
tolerance.

Why iterate? If you have sizable regions of missing pixels, you'll get
a fair bit of noise in the center since their values will be
controlled only by a few distance pixels at the edge of your kernel.
Just moving to the next missing pixel might get an entirely new set of
known pixels. Iterating spreads the "good" information from the edges
of the missing region into the center. This is roughly akin to solving
a PDE over the missing region using the known pixels as boundary
conditions. I have no particular references for this approach, but I
imagine you can dig up something in the literature about PDE-based
image processing.

This question came up at the first Open Training we held at Enthought.
Mark Marlett brought up the iteration approach. I pooh-poohed it at
the time, preferring the RBF analogy of the single-pass approach, but
if you need largish holes to be smoothed out, I think it should work
better.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From rod.stephenson at gmail.com  Thu Jan 15 22:24:29 2009
From: rod.stephenson at gmail.com (Rodney Stephenson)
Date: Fri, 16 Jan 2009 13:24:29 +1000
Subject: [Numpy-discussion] Long integer 'in' arange()
Message-ID: <49708A8D.2414.CD0ED76@rod.stephenson.transcritical.com>

There was some recent discussion about the (inefficiency) of 'in' when
applied to a large xrange. I was wondering how numpy handled this and
found the following

    >>> from numpy import *
    >>> u = arange(2**26)
    >>> 2**24 in u
    True
    >>> 2**31 in u
    False
    >>> 2**32 in u
    True

Hmmmm...

    >>> u = arange(10, 2**26)
    >>> 1 in u
    False
    >>> 2**37 in u
    False
    >>> 0 in u
    False
    >>> u[0]=0
    >>> 2**37 in u
    True
    >>> 

So I presume that in casting the long to int for the test is done by 
just taking the last 32 bits! Bug or feature? Python silently turns 
ints to longs when they overflow, so one should be careful...



From stefan at sun.ac.za  Thu Jan 15 23:37:13 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 16 Jan 2009 06:37:13 +0200
Subject: [Numpy-discussion] Help with interpolating missing values from
	a 3D scanner
In-Reply-To: <3d375d730901151614h2c3b8478y9900bba97dbd8381@mail.gmail.com>
References: <gknb08$pnq$1@ger.gmane.org>
	<EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
	<3d375d730901151614h2c3b8478y9900bba97dbd8381@mail.gmail.com>
Message-ID: <9457e7c80901152037v6d4138efq7a0ff5464b6b7c19@mail.gmail.com>

2009/1/16 Robert Kern <robert.kern at gmail.com>:
> of the missing region into the center. This is roughly akin to solving
> a PDE over the missing region using the known pixels as boundary
> conditions. I have no particular references for this approach, but I
> imagine you can dig up something in the literature about PDE-based
> image processing.

This technique is known as image inpainting:

http://www.iua.upf.es/~mbertalmio/restoration.html

Regards
St?fan


From cournape at gmail.com  Thu Jan 15 23:53:09 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 16 Jan 2009 13:53:09 +0900
Subject: [Numpy-discussion] remove need for site.cfg on default
In-Reply-To: <a08e5f80901150718ke9d0b38l80c7e2291be56867@mail.gmail.com>
References: <c7009a550901132321geac2de0o9a9e675aa9257510@mail.gmail.com>
	<a08e5f80901150718ke9d0b38l80c7e2291be56867@mail.gmail.com>
Message-ID: <5b8d13220901152053s5d1bf483jf3773d9ed17ef285@mail.gmail.com>

On Fri, Jan 16, 2009 at 12:18 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> Hi Jarrod,
>
> On Wed, Jan 14, 2009 at 2:21 AM, Jarrod Millman <millman at berkeley.edu>
> wrote:
>>
>> Due to the fact that I was tired of adding site.cfg to scipy and numpy
>> when building on Fedora and Ubuntu systems as well as a scipy ticket
>> (http://scipy.org/scipy/numpy/ticket/985), I decided to try and add
>> default system paths to numpy.distutils.  You can find out more
>> details on this ticket:
>>  http://projects.scipy.org/scipy/scipy/ticket/817
>>
>> I would like to have as many people test this as possible.  So I would
>> like everyone, who has had to build numpy/scipy with a site.cfg
>> despite having installed all the dependencies in the system default
>> locations, to test this.  If you would please update to the numpy
>> trunk and build numpy and scipy without your old site.cfg.  Regardless
>> of whether it works or not, I would appreciate it if you could let me
>> know.
>
> I tested on Gentoo, which does some non-standard stuff in an attempt to be
> able to switch between implementations. The libraries in /usr/lib are
> symlinks to the implementations in /usr/lib/blas/, and you can have various
> implementations like atlas, reference, or threaded-atlas, so I can run a
> program called eselect that lets me switch between the implementations. For
> some reason, gentoo renames the f77blas libs to simply blas.

That really baffles me. Why do distributions think it is ok to
randomly rename libraries - they certainly could handle a eselect
system without renaming libraries, so that they avoid breaking the
softwares depending on the libraries,

David


From scott.sinclair.za at gmail.com  Fri Jan 16 02:42:55 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Fri, 16 Jan 2009 09:42:55 +0200
Subject: [Numpy-discussion] Help with interpolating missing values from
	a 3D scanner
In-Reply-To: <3d375d730901151614h2c3b8478y9900bba97dbd8381@mail.gmail.com>
References: <gknb08$pnq$1@ger.gmane.org>
	<EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
	<3d375d730901151614h2c3b8478y9900bba97dbd8381@mail.gmail.com>
Message-ID: <6a17e9ee0901152342l533ec36cp24ac8918be1e3bf3@mail.gmail.com>

> 2009/1/16 Robert Kern <robert.kern at gmail.com>:
> On Thu, Jan 15, 2009 at 16:55, David Bolme <bolme1234 at comcast.net> wrote:
>>
>> I am working on a face recognition using 3D data from a special 3D
>> imaging system.  For those interested the data comes from the FRGC
>> 2004 dataset.  The problem I am having is that for some pixels the
>> scanner fails to capture depth information.  The result is that the
>> image has missing values.  There are small regions on the face such as
>> eyebrows and eyes that are missing the depth information.  I would
>> like to fill in these region by interpolating from nearby pixels but I
>> am not sure of the best way to do that.
>
> Another approach (that you would have to code yourself) is to take a
> Gaussian smoothing kernel of an appropriate size, center it over each
> missing pixel, then average the known pixels under the kernel using
> the kernel as a weighting factor. Place that average value into the
> missing pixel. This is actually fairly similar to the Rbf method
> above, but will probably be more efficient since you know that the
> points are all gridded.

You might try using Rbf with a window of known pixels centred on your
missing pixels. You'll automatically get a smoothing kernel that
weights nearer known pixel values more heavily, the behaviour of the
kernel depends on the basis function you choose (so it's similar to
the Gaussian smoothing idea). The reason for using a window is
efficiency, Rbf will be grossly inefficient if you feed it all of the
known pixels in your image as known values. Using a window will gain
efficiency without significantly changing your result because very
distant known pixel values contribute little to the result anyway.

The iteration of image inpainting also sounds like a useful extension.

Cheers,
Scott


From slaunger at gmail.com  Fri Jan 16 06:02:09 2009
From: slaunger at gmail.com (Kim Hansen)
Date: Fri, 16 Jan 2009 12:02:09 +0100
Subject: [Numpy-discussion] How to make "lazy" derived arrays in a recarray
	view of a memmap on large files
Message-ID: <f2e7a4240901160302w758afc31oe39a9952ca53c975@mail.gmail.com>

Hi numpy forum

I need to efficiently handle some large (300 MB) recordlike binary
files, where some data fields are less than a byte and thus cannot be
mapped in a record dtype immediately.

I would like to be able to access these derived arrays in a memory
efficient manner but I cannot figure out how to acheive this.

My application of the derived arrays would never be to do operation on
the entire array, rather iterate over some selected elements and do
somthing about it - operations which seems well suited for doing on
demand

I wrote a related post yesterday, which I have not received any
response on. I am now posting again using another example and perhaps
more clear example which I beleive describes my problem spot on

from numpy import *

# Python.exe memory use here: 8.14 MB
desc = dtype([("baconandeggs", "<u1"), ("spam","<u1"), ("parrots","<u1")])
index = memmap("g:/id-2008-10-25-17-ver4.idx", dtype = desc,
mode="r").view(recarray)
# The index file is very large, contains 292 MB of data
# Python.exe memory use: 8.16 MB, only 20 kB extra for memmap mapped to recarray

# The following instant operation takes a few secs working on 3*10^8 elements
# How can I derive new array in a lazy/ondemand/memmap manner?
index.bacon = index.baconandeggs >> 4
# python.exe memory use: 595 MB! Not surprising but how to do better??

# Another derived array, which is resource demanding
index.eggs = index.baconandeggs & 0x0F
# python.exe memory usage is now 731 MB!

What I'd like to do is implement a class, LazyBaconEggsSpamParrots,
which encapsulates the
derived arrays

such that I could do

besp = LazyBaconEggsSpamParrots("baconeggsspamparrots.idx")
for b in besp.bacon: #Iterate lazy
    spam(b)
#Only derive the 1000 needed elements, don't do all 1000000
dosomething(besp.bacon[1000000:1001000])

I envision the class would look something like this

class LazyBaconEggsSpamParrots(object):

    def __init__(self, filename):
         desc = dtype([("baconandeggs", "<u1"),
                       ("spam","<u1"),
                       ("parrots","<u1")])
         self._data = memmap(filename, dtype=desc, mode='r').view(recarray)
         # Expose the one-to-one data directly
         self.spam = self._data.spam
         self.parrots = self._data.parrots
         # This would work but costs way too much memory
         # self.bacon = self._data.baconandeggs >> 4
         # self.eggs = self._data.baconandeggs & 0x0F

    def __getattr__(self, attr_name):
        if attr_name == "bacon":
            # return bacon in an on demand manner, but how?
        elif attr_name == "eggs":
            # return eggs in an on demand manner, but how?
        else:
            # If the name is not a data attribute treat it as a normal
            # non-existing attribute - raise AttributeError
            raise AttributeError

but how to do the lazy part of it?

-- Kim


From faltet at pytables.org  Fri Jan 16 07:00:25 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 16 Jan 2009 13:00:25 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1, an efficient array evaluator
Message-ID: <200901161300.26130.faltet@pytables.org>

========================
 Announcing Numexpr 1.1
========================

Numexpr is a fast numerical expression evaluator for NumPy.  With it,
expressions that operate on arrays (like "3*a+4*b") are accelerated
and use less memory than doing the same calculation in Python.

The expected speed-ups for Numexpr respect to NumPy are between 0.95x
and 15x, being 3x or 4x typical values.  The strided and unaligned
case has been optimized too, so if the expresion contains such arrays, 
the speed-up can increase significantly.  Of course, you will need to 
operate with large arrays (typically larger than the cache size of your 
CPU) to see these improvements in performance.

This release is mainly intended to put in sync some of the
improvements that had the Numexpr version integrated in PyTables.
So, this standalone version of Numexpr will benefit of the well tested 
PyTables' version that has been in production for more than a year now.
  
In case you want to know more in detail what has changed in this
version, have a look at ``RELEASE_NOTES.txt`` in the tarball.


Where I can find Numexpr?
=========================

The project is hosted at Google code in:

http://code.google.com/p/numexpr/


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.


Enjoy!

-- 
Francesc Alted


From dsdale24 at gmail.com  Fri Jan 16 08:07:08 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 16 Jan 2009 08:07:08 -0500
Subject: [Numpy-discussion] remove need for site.cfg on default
In-Reply-To: <5b8d13220901152053s5d1bf483jf3773d9ed17ef285@mail.gmail.com>
References: <c7009a550901132321geac2de0o9a9e675aa9257510@mail.gmail.com>
	<a08e5f80901150718ke9d0b38l80c7e2291be56867@mail.gmail.com>
	<5b8d13220901152053s5d1bf483jf3773d9ed17ef285@mail.gmail.com>
Message-ID: <a08e5f80901160507s78f98da0q70834e6f59259533@mail.gmail.com>

On Thu, Jan 15, 2009 at 11:53 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Fri, Jan 16, 2009 at 12:18 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> > Hi Jarrod,
> >
> > On Wed, Jan 14, 2009 at 2:21 AM, Jarrod Millman <millman at berkeley.edu>
> > wrote:
> >>
> >> Due to the fact that I was tired of adding site.cfg to scipy and numpy
> >> when building on Fedora and Ubuntu systems as well as a scipy ticket
> >> (http://scipy.org/scipy/numpy/ticket/985), I decided to try and add
> >> default system paths to numpy.distutils.  You can find out more
> >> details on this ticket:
> >>  http://projects.scipy.org/scipy/scipy/ticket/817
> >>
> >> I would like to have as many people test this as possible.  So I would
> >> like everyone, who has had to build numpy/scipy with a site.cfg
> >> despite having installed all the dependencies in the system default
> >> locations, to test this.  If you would please update to the numpy
> >> trunk and build numpy and scipy without your old site.cfg.  Regardless
> >> of whether it works or not, I would appreciate it if you could let me
> >> know.
> >
> > I tested on Gentoo, which does some non-standard stuff in an attempt to
> be
> > able to switch between implementations. The libraries in /usr/lib are
> > symlinks to the implementations in /usr/lib/blas/, and you can have
> various
> > implementations like atlas, reference, or threaded-atlas, so I can run a
> > program called eselect that lets me switch between the implementations.
> For
> > some reason, gentoo renames the f77blas libs to simply blas.
>
> That really baffles me. Why do distributions think it is ok to
> randomly rename libraries - they certainly could handle a eselect
> system without renaming libraries, so that they avoid breaking the
> softwares depending on the libraries,
>

I am in complete agreement. I've raised the issue with them before and didnt
make any headway. Maybe its time to try again.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090116/818f64fd/attachment.html>

From pearu at cens.ioc.ee  Fri Jan 16 08:16:18 2009
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Fri, 16 Jan 2009 15:16:18 +0200 (EET)
Subject: [Numpy-discussion] error handling with f2py?
In-Reply-To: <496F61AD.8080902@molden.no>
References: <496F61AD.8080902@molden.no>
Message-ID: <41228.172.17.0.4.1232111778.squirrel@cens.ioc.ee>

On Thu, January 15, 2009 6:17 pm, Sturla Molden wrote:
> Is it possible to make f2py raise an exception if a fortran routine
> signals an error?
>
> If I e.g. have
>
>      subroutine foobar(a, ierr)
>
> Can I get an exception automatically raised if ierr != 0?

Yes, for that you need to provide your own fortran call code
using f2py callstatement construct. The initial fortran call
code can be obtained from f2py generated <modulename>module.c file,
for instance.

An example follows below:

Fortran file foo.f:
-------------------

      subroutine foo(a, ierr)
      integer a
      integer ierr
      if (a.gt.10) then
        ierr=2
      else
         if (a.gt.5) then
            ierr=1
         else
            ierr = 0
         end if
      end if
      end

Generated (f2py -m m foo.f) and then modified signature file m.pyf:
-------------------------------------------------------------------

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module m ! in
    interface  ! in :m
        subroutine foo(a,ierr) ! in :m:foo.f
            integer :: a
            integer :: ierr
            intent (in, out) a
            intent (hide) ierr
            callstatement '''
            (*f2py_func)(&a, &ierr);
            if (ierr==1)
            {
              PyErr_SetString(PyExc_ValueError, "a is gt 5");
              }
            if (ierr==2)
              {
                PyErr_SetString(PyExc_ValueError, "a is gt 10");
              }
            '''
        end subroutine foo
    end interface
end python module m

! This file was auto-generated with f2py (version:2_5618).
! See http://cens.ioc.ee/projects/f2py2e/

Build the extension module and use from python:
-----------------------------------------------

$ f2py -c m.pyf foo.f
$ python
>>> import m
>>> m.foo(30)
---------------------------------------------------------------------------
<type 'exceptions.ValueError'>            Traceback (most recent call last)

/home/pearu/test/f2py/exc/<ipython console> in <module>()

<type 'exceptions.ValueError'>: a is gt 10
>>> m.foo(6)
---------------------------------------------------------------------------
<type 'exceptions.ValueError'>            Traceback (most recent call last)

/home/pearu/test/f2py/exc/<ipython console> in <module>()

<type 'exceptions.ValueError'>: a is gt 5
>>> m.foo(4)
4

HTH,
Pearu





From gwg at emss.co.za  Fri Jan 16 09:07:48 2009
From: gwg at emss.co.za (George)
Date: Fri, 16 Jan 2009 14:07:48 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?Singular_Matrix_problem_with_Matplit?=
	=?utf-8?q?lib_in=09Numpy_=28Windows_-_AMD64=29?=
References: <15B34CD0955E484689D667626E6456D5011C8E787E@london.emss.co.za>	<5b8d13220812080943g69d4c670jabd6aef66d336e29@mail.gmail.com>
	<loom.20081217T094146-465@post.gmane.org>
	<4948D139.4020703@gmail.com>
	<loom.20081217T182123-184@post.gmane.org>
	<loom.20090115T192826-15@post.gmane.org>
Message-ID: <loom.20090116T134622-110@post.gmane.org>

Hello.

I am terribly sorry. I was mistaken last night. I had the latest Matplotlib
version 0.98.5.2 and I thought the bug was fixed but it hasn't. Let me explain.

In the file MPL_isnan.h line 26 there is a declaration:

typedef long int                             MPL_Int64

This is fine for Linux 64-bit, but NOT for Windows XP 64-bit! For Windows the 
declaration should be:

typedef long long                            MPL_Int64

This bug has caused me a LOT of late nights and last night was one of them. The
declaration is correct for Linux 64-bit and I guess Matplotlib was developed on 
Linux because of this declaration. That is also why I thought the bug was fixed
but this morning I realised that I was looking at the wrong console.

So, in summary. For Matplotlib 0.98.5.2 and Numpy 1.2.1 to work without any 
problems. This means compiling and using Numpy and Matplotlib on Windows XP 
64-bit using AMD 64-bit compile environment, change line 26 in the file
MPL_isnan.h from long int to long long.\

I also previously suggested switching MKL and ACML etc. but with this change
everything is fine. One can choose any math library and it works.

Writing a small test application using sizeof on different platforms highlights
the problem.

Thanks.

George.




From jdh2358 at gmail.com  Fri Jan 16 09:16:18 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Fri, 16 Jan 2009 08:16:18 -0600
Subject: [Numpy-discussion] Singular Matrix problem with Matplitlib in
	Numpy (Windows - AMD64)
In-Reply-To: <loom.20090116T134622-110@post.gmane.org>
References: <15B34CD0955E484689D667626E6456D5011C8E787E@london.emss.co.za>
	<5b8d13220812080943g69d4c670jabd6aef66d336e29@mail.gmail.com>
	<loom.20081217T094146-465@post.gmane.org> <4948D139.4020703@gmail.com>
	<loom.20081217T182123-184@post.gmane.org>
	<loom.20090115T192826-15@post.gmane.org>
	<loom.20090116T134622-110@post.gmane.org>
Message-ID: <88e473830901160616t3fb33e3v826ebaa395062cf5@mail.gmail.com>

Andrew, since you are the original author of the isnan port, could you
patch the branch and the trunk to take care of this?

JDH

On Fri, Jan 16, 2009 at 8:07 AM, George <gwg at emss.co.za> wrote:
> Hello.
>
> I am terribly sorry. I was mistaken last night. I had the latest Matplotlib
> version 0.98.5.2 and I thought the bug was fixed but it hasn't. Let me explain.
>
> In the file MPL_isnan.h line 26 there is a declaration:
>
> typedef long int                             MPL_Int64
>
> This is fine for Linux 64-bit, but NOT for Windows XP 64-bit! For Windows the
> declaration should be:
>
> typedef long long                            MPL_Int64
>
> This bug has caused me a LOT of late nights and last night was one of them. The
> declaration is correct for Linux 64-bit and I guess Matplotlib was developed on
> Linux because of this declaration. That is also why I thought the bug was fixed
> but this morning I realised that I was looking at the wrong console.
>
> So, in summary. For Matplotlib 0.98.5.2 and Numpy 1.2.1 to work without any
> problems. This means compiling and using Numpy and Matplotlib on Windows XP
> 64-bit using AMD 64-bit compile environment, change line 26 in the file
> MPL_isnan.h from long int to long long.\
>
> I also previously suggested switching MKL and ACML etc. but with this change
> everything is fine. One can choose any math library and it works.
>
> Writing a small test application using sizeof on different platforms highlights
> the problem.
>
> Thanks.
>
> George.
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From gregor.thalhammer at gmail.com  Fri Jan 16 09:39:57 2009
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Fri, 16 Jan 2009 15:39:57 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161300.26130.faltet@pytables.org>
References: <200901161300.26130.faltet@pytables.org>
Message-ID: <49709C3D.1010608@googlemail.com>

Francesc Alted schrieb:
> Numexpr is a fast numerical expression evaluator for NumPy.  With it,
> expressions that operate on arrays (like "3*a+4*b") are accelerated
> and use less memory than doing the same calculation in Python.
>
> The expected speed-ups for Numexpr respect to NumPy are between 0.95x
> and 15x, being 3x or 4x typical values.  The strided and unaligned
> case has been optimized too, so if the expresion contains such arrays, 
> the speed-up can increase significantly.  Of course, you will need to 
> operate with large arrays (typically larger than the cache size of your 
> CPU) to see these improvements in performance.
>
>   
Just recently I had a more detailed look at numexpr. Clever idea, easy 
to use! I can affirm an typical performance gain of 3x if you work on 
large arrays (>100k entries).

I also gave a try to the vector math library (VML), contained in Intel's 
Math Kernel Library. This offers a fast implementation of mathematical 
functions, operating on array. First I implemented a C extension, 
providing new ufuncs. This gave me a big performance gain, e.g., 2.3x 
(5x) for sin, 6x (10x) for exp, 7x (15x) for pow, and 3x (6x) for 
division (no gain for add, sub, mul). The values in parantheses are 
given if I allow VML to use several threads and to employ both cores of 
my Intel Core2Duo computer. For large arrays (100M entries) this 
performance gain is reduced because of limited memory bandwidth. At this 
point I was stumbling across numexpr and modified it to use the VML 
functions. For sufficiently long and complex  numerical expressions I 
could  get the maximum performance also for large arrays.  Together with 
VML numexpr seems to be a extremely powerful to get an optimum 
performance. I would like to see numexpr extended to (optionally) make 
use of fast vectorized math functions. There is one but: VML supports 
(at the moment) only math on contiguous arrays. At a first try I didn't 
understand how to enforce this limitation in numexpr. I also gave a 
quick try to the equivalent vector math library, acml_mv of AMD. I only 
tried sin and log, gave me the same performance (on a Intel processor!) 
like Intels VML .

I was also playing around with the block size in numexpr. What are the 
rationale that led to the current block size of 128? Especially with 
VML, a larger block size of 4096 instead of 128 allowed  to efficiently 
use multithreading in VML.
> Share your experience
> =====================
>
> Let us know of any bugs, suggestions, gripes, kudos, etc. you may
> have.
>
>   
I was missing the support for single precision floats.

Great work!

Gregor


From josef.pktd at gmail.com  Fri Jan 16 10:51:19 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 16 Jan 2009 10:51:19 -0500
Subject: [Numpy-discussion] numpy.testing.asserts and masked array
Message-ID: <1cd32cbb0901160751l235bd4dfudfaaa3d221346214@mail.gmail.com>

I have a regression result with masked arrays that produces a masked
array output, estm5.yhat, and I want to test equality to the benchmark
case, estm1.yhat, with the asserts in numpy.testing, but I am getting
strange results.

checking for equality directly returns True, but the asserts in
numpy.testing think that they are not equal. It seems the asserts use
one of the masked (random) values in the comparison.

Whats the trick to assert_almost_equal for masked arrays?

Josef

>>> np.all(estm5.yhat == estm1.yhat)
True

and they look the same:
>>> estm5.yhat.T
masked_array(data =
 [[-- 3.72175928422 4.44050187015 3.86191008624 7.54173762712 3.52929761101
  5.00953694359 5.19214323965 4.71500145194 8.22336216275 7.8884375775
  5.24556560163 6.81814135969 7.65416592877 3.7608331097]],
      mask =
 [[ True False False False False False False False False False False False
  False False False]],
      fill_value=1e+020)

>>> estm1.yhat.T
masked_array(data =
 [[-- 3.72175928422 4.44050187015 3.86191008624 7.54173762712 3.52929761101
  5.00953694359 5.19214323965 4.71500145194 8.22336216275 7.8884375775
  5.24556560163 6.81814135969 7.65416592877 3.7608331097]],
      mask =
 [[ True False False False False False False False False False False False
  False False False]],
      fill_value=1e+020)




using any np.testing.assert returns False

>>> np.testing.assert_array_almost_equal(estm5.yhat, estm1.yhat)
Traceback (most recent call last):
  File "<pyshell#253>", line 1, in <module>
    np.testing.assert_array_almost_equal(estm5.yhat, estm1.yhat)
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 310, in assert_array_almost_equal
    header='Arrays are not almost equal')
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 299, in assert_array_compare
    raise ValueError(msg)
ValueError:
Arrays are not almost equal
 x: array([[ 6.1104881 ],
       [ 3.72175928],
       [ 4.44050187],...
 y: array([[        NaN],
       [ 3.72175928],
       [ 4.44050187],...
>>> np.testing.assert_almost_equal(estm5.yhat, estm1.yhat)
Traceback (most recent call last):
  File "<pyshell#254>", line 1, in <module>
    np.testing.assert_almost_equal(estm5.yhat, estm1.yhat)
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 208, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 310, in assert_array_almost_equal
    header='Arrays are not almost equal')
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 299, in assert_array_compare
    raise ValueError(msg)
ValueError:
Arrays are not almost equal
 x: array([[ 6.1104881 ],
       [ 3.72175928],
       [ 4.44050187],...
 y: array([[        NaN],
       [ 3.72175928],
       [ 4.44050187],...
>>> np.testing.assert_equal(estm5.yhat, estm1.yhat)
Traceback (most recent call last):
  File "<pyshell#255>", line 1, in <module>
    np.testing.assert_equal(estm5.yhat, estm1.yhat)
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 180, in assert_equal
    return assert_array_equal(actual, desired, err_msg, verbose)
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 303, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py",
line 295, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 100.0%)
 x: array([[ 6.1104881 ],
       [ 3.72175928],
       [ 4.44050187],...
 y: array([[        NaN],
       [ 3.72175928],
       [ 4.44050187],...
>>> estm5.yhat == estm1.yhat
masked_array(data =
 [[--]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]
 [True]],
      mask =
 [[ True]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]
 [False]],
      fill_value=1e+020)


From pgmdevlist at gmail.com  Fri Jan 16 10:59:22 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 16 Jan 2009 10:59:22 -0500
Subject: [Numpy-discussion] numpy.testing.asserts and masked array
In-Reply-To: <1cd32cbb0901160751l235bd4dfudfaaa3d221346214@mail.gmail.com>
References: <1cd32cbb0901160751l235bd4dfudfaaa3d221346214@mail.gmail.com>
Message-ID: <CB012C2C-8353-4E15-888F-0A66B3055946@gmail.com>


On Jan 16, 2009, at 10:51 AM, josef.pktd at gmail.com wrote:

> I have a regression result with masked arrays that produces a masked
> array output, estm5.yhat, and I want to test equality to the benchmark
> case, estm1.yhat, with the asserts in numpy.testing, but I am getting
> strange results.

> ...
> Whats the trick to assert_almost_equal for masked arrays?

Us numpy.ma.testutils.assert_almost_equal instead.


From faltet at pytables.org  Fri Jan 16 11:04:51 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 16 Jan 2009 17:04:51 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
Message-ID: <200901161704.51777.faltet@pytables.org>

A Friday 16 January 2009, jh at physics.ucf.edu escrigu?:
> Hi Francesc,
>
> > Numexpr is a fast numerical expression evaluator for NumPy.  With
> > it, expressions that operate on arrays (like "3*a+4*b") are
> > accelerated and use less memory than doing the same calculation in
> > Python.
>
> Please pardon my ignorance as I know this project has been around for
> a while.  It this looks very exciting, but either it's cumbersome, or
> I'm not understanding exactly what's being fixed.  If you can
> accelerate evaluation, why not just integrate the faster math into
> numpy, rather than having two packages?  Or is this something that is
> only an advantage when the expression is given as a string (and why
> is that the case)?  It would be helpful if you could put the answer
> on your web page and in your standard release blurb in some compact
> form. I guess what I'm really looking for when I read one of those is
> a quick answer to the question "should I look into this?".

Well, there is a link in the project page to the "Overview" section of 
the wiki, but perhaps is a bit hidden.  I've added some blurb as you 
suggested in the main page an another link to the "Overview" wiki page.
Hope that, by reading the new blurb, you can see why it accelerates 
expression evaluation with regard to NumPy.  If not, tell me and will 
try to come with something more comprehensible.

> Right 
> now, I'm not quite sure whether the problem you are solving is merely
> the case of expressions-in-strings, and there is no advantage for
> expressions-in-code, or whether your expressions-in-strings are
> faster than numpy's expressions-in-code. In either case, it would 
> appear this would be a good addition to the numpy core, and it's past
> 1.0, so why keep it separate?  Even if there is value in having a
> non-numpy version, is there not also value in accelerating numpy by
> default?

Having the expression encapsulated in a string has the advantage that 
you exactly know the part of the code that you want to parse and 
accelerate.  Making NumPy to understand parts of the Python code that 
can be accelerated sounds more like a true JIT for Python, and this is 
something that is not trivial at all (although, with the advent of PyPy 
there are appearing some efforts in this direction [1]).

[1] http://www.enthought.com/~ischnell/paper.html

Cheers,

-- 
Francesc Alted


From sturla at molden.no  Fri Jan 16 11:15:06 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 16 Jan 2009 17:15:06 +0100
Subject: [Numpy-discussion] error handling with f2py?
In-Reply-To: <41228.172.17.0.4.1232111778.squirrel@cens.ioc.ee>
References: <496F61AD.8080902@molden.no>
	<41228.172.17.0.4.1232111778.squirrel@cens.ioc.ee>
Message-ID: <4970B28A.6010106@molden.no>

On 1/16/2009 2:16 PM, Pearu Peterson wrote:

> Yes, for that you need to provide your own fortran call code
> using f2py callstatement construct. The initial fortran call
> code can be obtained from f2py generated <modulename>module.c file,
> for instance.

Thank you, Pearu :)

f2py is really a wonderful tool.


Sturla Molden


From haase at msg.ucsf.edu  Fri Jan 16 11:20:03 2009
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Fri, 16 Jan 2009 17:20:03 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161704.51777.faltet@pytables.org>
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
	<200901161704.51777.faltet@pytables.org>
Message-ID: <bc657ead0901160820n7dd66ed3nd7502ee34a05680f@mail.gmail.com>

Hi Francesc,
this is a wonderful project ! I was just wondering if you would /
could support single precision float arrays ?
In 3+D image analysis we generally don't have enough memory to effort
double precision; and we could save our selves lots of extra C coding
(or Cython) coding of we could use numexpr ;-)

Thanks,
Sebastian Haase



On Fri, Jan 16, 2009 at 5:04 PM, Francesc Alted <faltet at pytables.org> wrote:
> A Friday 16 January 2009, jh at physics.ucf.edu escrigu?:
>> Hi Francesc,
>>
>> > Numexpr is a fast numerical expression evaluator for NumPy.  With
>> > it, expressions that operate on arrays (like "3*a+4*b") are
>> > accelerated and use less memory than doing the same calculation in
>> > Python.
>>
>> Please pardon my ignorance as I know this project has been around for
>> a while.  It this looks very exciting, but either it's cumbersome, or
>> I'm not understanding exactly what's being fixed.  If you can
>> accelerate evaluation, why not just integrate the faster math into
>> numpy, rather than having two packages?  Or is this something that is
>> only an advantage when the expression is given as a string (and why
>> is that the case)?  It would be helpful if you could put the answer
>> on your web page and in your standard release blurb in some compact
>> form. I guess what I'm really looking for when I read one of those is
>> a quick answer to the question "should I look into this?".
>
> Well, there is a link in the project page to the "Overview" section of
> the wiki, but perhaps is a bit hidden.  I've added some blurb as you
> suggested in the main page an another link to the "Overview" wiki page.
> Hope that, by reading the new blurb, you can see why it accelerates
> expression evaluation with regard to NumPy.  If not, tell me and will
> try to come with something more comprehensible.
>
>> Right
>> now, I'm not quite sure whether the problem you are solving is merely
>> the case of expressions-in-strings, and there is no advantage for
>> expressions-in-code, or whether your expressions-in-strings are
>> faster than numpy's expressions-in-code. In either case, it would
>> appear this would be a good addition to the numpy core, and it's past
>> 1.0, so why keep it separate?  Even if there is value in having a
>> non-numpy version, is there not also value in accelerating numpy by
>> default?
>
> Having the expression encapsulated in a string has the advantage that
> you exactly know the part of the code that you want to parse and
> accelerate.  Making NumPy to understand parts of the Python code that
> can be accelerated sounds more like a true JIT for Python, and this is
> something that is not trivial at all (although, with the advent of PyPy
> there are appearing some efforts in this direction [1]).
>
> [1] http://www.enthought.com/~ischnell/paper.html
>
> Cheers,
>
> --
> Francesc Alted
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From josef.pktd at gmail.com  Fri Jan 16 11:28:08 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 16 Jan 2009 11:28:08 -0500
Subject: [Numpy-discussion] numpy.testing.asserts and masked array
In-Reply-To: <CB012C2C-8353-4E15-888F-0A66B3055946@gmail.com>
References: <1cd32cbb0901160751l235bd4dfudfaaa3d221346214@mail.gmail.com>
	<CB012C2C-8353-4E15-888F-0A66B3055946@gmail.com>
Message-ID: <1cd32cbb0901160828y23252234s2dd34f9cf123f062@mail.gmail.com>

On Fri, Jan 16, 2009 at 10:59 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Jan 16, 2009, at 10:51 AM, josef.pktd at gmail.com wrote:
>
>> I have a regression result with masked arrays that produces a masked
>> array output, estm5.yhat, and I want to test equality to the benchmark
>> case, estm1.yhat, with the asserts in numpy.testing, but I am getting
>> strange results.
>
>> ...
>> Whats the trick to assert_almost_equal for masked arrays?
>
> Us numpy.ma.testutils.assert_almost_equal instead.

Thanks, when I looked at the test files for mstats, I forgot to check
the imports.

Josef


From faltet at pytables.org  Fri Jan 16 11:48:34 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 16 Jan 2009 17:48:34 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <49709C3D.1010608@googlemail.com>
References: <200901161300.26130.faltet@pytables.org>
	<49709C3D.1010608@googlemail.com>
Message-ID: <200901161748.34558.faltet@pytables.org>

A Friday 16 January 2009, Gregor Thalhammer escrigu?:
> I also gave a try to the vector math library (VML), contained in
> Intel's Math Kernel Library. This offers a fast implementation of
> mathematical functions, operating on array. First I implemented a C
> extension, providing new ufuncs. This gave me a big performance gain,
> e.g., 2.3x (5x) for sin, 6x (10x) for exp, 7x (15x) for pow, and 3x
> (6x) for division (no gain for add, sub, mul).

Wow, pretty nice speed-ups indeed!  In fact I was thinking in including 
support for threading in Numexpr (I don't think it would be too 
difficult, but let's see).  BTW, do you know how VML is able to achieve 
a speedup of 6x for a sin() function?  I suppose this is because they 
are using SSE instructions, but, are these also available for 64-bit 
double precision items?

> The values in 
> parantheses are given if I allow VML to use several threads and to
> employ both cores of my Intel Core2Duo computer. For large arrays
> (100M entries) this performance gain is reduced because of limited
> memory bandwidth. At this point I was stumbling across numexpr and
> modified it to use the VML functions. For sufficiently long and
> complex  numerical expressions I could  get the maximum performance
> also for large arrays.

Cool.

> Together with VML numexpr seems to be a 
> extremely powerful to get an optimum performance. I would like to see
> numexpr extended to (optionally) make use of fast vectorized math
> functions.

Well, if you can provide the code, I'd be glad to include it in numexpr.  
The only requirement is that the VML must be optional during the build 
of the package.

> There is one but: VML supports (at the moment) only math 
> on contiguous arrays. At a first try I didn't understand how to
> enforce this limitation in numexpr.

No problem.  At the end of the numexpr/necompiler.py you will see some 
code like:

        # All the opcodes can deal with strided arrays directly as
        # long as they are undimensional (strides in other
        # dimensions are dealt within the extension), so we don't
        # need a copy for the strided case.
        if not b.flags.aligned:
           ...

which you can replace with something like:

        # need a copy for the strided case.
        if VML_available and not b.flags.contiguous: 
	    b = b.copy()
        elif not b.flags.aligned:
	    ...

That would be enough for ensuring that all the arrays are contiguous 
when they hit numexpr's virtual machine.

Being said this, it is a shame that VML does not have support for 
strided/unaligned arrays.  They are quite common beasts, specially when 
you work with heterogeneous arrays (aka record arrays).

> I also gave a quick try to the 
> equivalent vector math library, acml_mv of AMD. I only tried sin and
> log, gave me the same performance (on a Intel processor!) like Intels
> VML .
>
> I was also playing around with the block size in numexpr. What are
> the rationale that led to the current block size of 128? Especially
> with VML, a larger block size of 4096 instead of 128 allowed  to
> efficiently use multithreading in VML.

Experimentation.  Back in 2006 David found that 128 was optimal for the 
processors available by that time.  With the Numexpr 1.1 my experiments 
show that 256 is a better value for current Core2 processors and most 
expressions in our benchmark bed (see benchmarks/ directory); hence, 
256 is the new value for the chunksize in 1.1.  However, be in mind 
that 256 has to be multiplied by the itemsize of each array, so the 
chunksize is currently 2048 bytes for 64-bit items (int64 or float64) 
and 4096 for double precision complex arrays, which are probably the 
sizes that have to be compared with VML.

>
> > Share your experience
> > =====================
> >
> > Let us know of any bugs, suggestions, gripes, kudos, etc. you may
> > have.

> I was missing the support for single precision floats.

Yeah.  This is because nobody has implemented it before, but it is 
completely doable.

> Great work!

You are welcome!  And thanks for excellent feedback too!  Hope we can 
have a VML-aware numexpr anytime soon ;-)

Cheers,

-- 
Francesc Alted


From faltet at pytables.org  Fri Jan 16 12:00:26 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 16 Jan 2009 18:00:26 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <bc657ead0901160820n7dd66ed3nd7502ee34a05680f@mail.gmail.com>
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
	<200901161704.51777.faltet@pytables.org>
	<bc657ead0901160820n7dd66ed3nd7502ee34a05680f@mail.gmail.com>
Message-ID: <200901161800.26988.faltet@pytables.org>

A Friday 16 January 2009, Sebastian Haase escrigu?:
> Hi Francesc,
> this is a wonderful project ! I was just wondering if you would /
> could support single precision float arrays ?

As I said before, it is doable, but I don't know if I will have time 
enough to implement this myself.

> In 3+D image analysis we generally don't have enough memory to effort
> double precision; and we could save our selves lots of extra C coding
> (or Cython) coding of we could use numexpr ;-)

Well, one of the ideas that I'm toying long time ago is to provide the 
capability to Numexpr to work with PyTables disk-based objects.  That 
way, you would be able to evaluate potentially complex expressions by 
using data that is completely on-disk.  But this might be a completely 
different thing of what you are talking about.

Cheers,

-- 
Francesc Alted


From dagss at student.matnat.uio.no  Fri Jan 16 12:35:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 16 Jan 2009 18:35:59 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161704.51777.faltet@pytables.org>
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
	<200901161704.51777.faltet@pytables.org>
Message-ID: <4970C57F.9030007@student.matnat.uio.no>

Francesc Alted wrote:
> A Friday 16 January 2009, jh at physics.ucf.edu escrigu?:
>> Right 
>> now, I'm not quite sure whether the problem you are solving is merely
>> the case of expressions-in-strings, and there is no advantage for
>> expressions-in-code, or whether your expressions-in-strings are
>> faster than numpy's expressions-in-code. In either case, it would 
>> appear this would be a good addition to the numpy core, and it's past
>> 1.0, so why keep it separate?  Even if there is value in having a
>> non-numpy version, is there not also value in accelerating numpy by
>> default?
> 
> Having the expression encapsulated in a string has the advantage that 
> you exactly know the part of the code that you want to parse and 
> accelerate.  Making NumPy to understand parts of the Python code that 
> can be accelerated sounds more like a true JIT for Python, and this is 
> something that is not trivial at all (although, with the advent of PyPy 
> there are appearing some efforts in this direction [1]).

A full compiler/JIT isn't needed, there's another route:

One could use the Numexpr methodology together with a symbolic 
expression framework (like SymPy or the one in Sage). I.e. operator 
overloads and lazy expressions.

Combining NumExpr with a symbolic manipulation engine would be very cool 
IMO. Unfortunately I don't have time myself (and I understand that you 
don't, I'm just mentioning it).

Example using psuedo-Sage-like syntax:

a = np.arange(bignum)
b = np.arange(bignum)
x, y = sage.var("x, y")
expr = sage.integrate(x + y, x)
z = expr(x=a, y=b) # z = a**2/2 + b, but Numexpr-enabled

-- 
Dag Sverre


From strawman at astraw.com  Fri Jan 16 13:38:01 2009
From: strawman at astraw.com (Andrew Straw)
Date: Fri, 16 Jan 2009 10:38:01 -0800
Subject: [Numpy-discussion] Singular Matrix problem with Matplitlib in
 Numpy (Windows - AMD64)
In-Reply-To: <88e473830901160616t3fb33e3v826ebaa395062cf5@mail.gmail.com>
References: <15B34CD0955E484689D667626E6456D5011C8E787E@london.emss.co.za>	
	<5b8d13220812080943g69d4c670jabd6aef66d336e29@mail.gmail.com>	
	<loom.20081217T094146-465@post.gmane.org>
	<4948D139.4020703@gmail.com>	
	<loom.20081217T182123-184@post.gmane.org>	
	<loom.20090115T192826-15@post.gmane.org>	
	<loom.20090116T134622-110@post.gmane.org>
	<88e473830901160616t3fb33e3v826ebaa395062cf5@mail.gmail.com>
Message-ID: <4970D409.8010805@astraw.com>

John Hunter wrote:
> Andrew, since you are the original author of the isnan port, could you
> patch the branch and the trunk to take care of this?

Done in r6791 and r6792.

Sorry for the trouble.

Now I just hope we don't get a problem with "long long", although now if
_ISOC99_SOURCE is defined, we'll preferentially use "int64_t" out of
<stdint.h>, so I should think this is more portable on sane platforms.

This one of many reasons why I stick to Python...
-Andrew

> 
> JDH
> 
> On Fri, Jan 16, 2009 at 8:07 AM, George <gwg at emss.co.za> wrote:
>> Hello.
>>
>> I am terribly sorry. I was mistaken last night. I had the latest Matplotlib
>> version 0.98.5.2 and I thought the bug was fixed but it hasn't. Let me explain.
>>
>> In the file MPL_isnan.h line 26 there is a declaration:
>>
>> typedef long int                             MPL_Int64
>>
>> This is fine for Linux 64-bit, but NOT for Windows XP 64-bit! For Windows the
>> declaration should be:
>>
>> typedef long long                            MPL_Int64
>>
>> This bug has caused me a LOT of late nights and last night was one of them. The
>> declaration is correct for Linux 64-bit and I guess Matplotlib was developed on
>> Linux because of this declaration. That is also why I thought the bug was fixed
>> but this morning I realised that I was looking at the wrong console.
>>
>> So, in summary. For Matplotlib 0.98.5.2 and Numpy 1.2.1 to work without any
>> problems. This means compiling and using Numpy and Matplotlib on Windows XP
>> 64-bit using AMD 64-bit compile environment, change line 26 in the file
>> MPL_isnan.h from long int to long long.\
>>
>> I also previously suggested switching MKL and ACML etc. but with this change
>> everything is fine. One can choose any math library and it works.
>>
>> Writing a small test application using sizeof on different platforms highlights
>> the problem.
>>
>> Thanks.
>>
>> George.
>>
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> 



From gregor.thalhammer at gmail.com  Fri Jan 16 14:35:37 2009
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Fri, 16 Jan 2009 20:35:37 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161748.34558.faltet@pytables.org>
References: <200901161300.26130.faltet@pytables.org>
	<49709C3D.1010608@googlemail.com>
	<200901161748.34558.faltet@pytables.org>
Message-ID: <4970E189.8000906@googlemail.com>

Francesc Alted schrieb:
> A Friday 16 January 2009, Gregor Thalhammer escrigu?:
>   
>> I also gave a try to the vector math library (VML), contained in
>> Intel's Math Kernel Library. This offers a fast implementation of
>> mathematical functions, operating on array. First I implemented a C
>> extension, providing new ufuncs. This gave me a big performance gain,
>> e.g., 2.3x (5x) for sin, 6x (10x) for exp, 7x (15x) for pow, and 3x
>> (6x) for division (no gain for add, sub, mul).
>>     
>
> Wow, pretty nice speed-ups indeed!  In fact I was thinking in including 
> support for threading in Numexpr (I don't think it would be too 
> difficult, but let's see).  BTW, do you know how VML is able to achieve 
> a speedup of 6x for a sin() function?  I suppose this is because they 
> are using SSE instructions, but, are these also available for 64-bit 
> double precision items?
>   
I am not an expert on SSE instructions, but to my knowledge there exist 
(in the Core 2 architecture) no SSE instruction to calculate the sin. 
But it seems to be possible to (approximately) calculate a sin with a 
couple of multiplication/ addition instructions (and they exist in SSE 
for 64-bit float). Intel (and AMD) seems to use a more clever algorithm, 
efficiently implemented than the standard implementation.
> Well, if you can provide the code, I'd be glad to include it in numexpr.  
> The only requirement is that the VML must be optional during the build 
> of the package.
>   
Yes, I will try to provide you with a polished version of my changes, 
making them optional.
>   
>> There is one but: VML supports (at the moment) only math 
>> on contiguous arrays. At a first try I didn't understand how to
>> enforce this limitation in numexpr.
>>     
>
> No problem.  At the end of the numexpr/necompiler.py you will see some 
> code like:
>
>         # All the opcodes can deal with strided arrays directly as
>         # long as they are undimensional (strides in other
>         # dimensions are dealt within the extension), so we don't
>         # need a copy for the strided case.
>         if not b.flags.aligned:
>            ...
>
> which you can replace with something like:
>
>         # need a copy for the strided case.
>         if VML_available and not b.flags.contiguous: 
> 	    b = b.copy()
>         elif not b.flags.aligned:
> 	    ...
>
> That would be enough for ensuring that all the arrays are contiguous 
> when they hit numexpr's virtual machine.
>   
Ah I see, that's not difficult. I thought copying is done in the virtual 
machine. (didn't read all the code ...)
> Being said this, it is a shame that VML does not have support for 
> strided/unaligned arrays.  They are quite common beasts, specially when 
> you work with heterogeneous arrays (aka record arrays).
>   
I have the impression that you can already feel happy if these 
mathematical libraries support a C interface, not only Fortran. At least 
the Intel VML provides functions to pack/unpack strided arrays which 
seem work on a broader parameter range than specified (also zero or 
negative step sizes).
>> I also gave a quick try to the 
>> equivalent vector math library, acml_mv of AMD. I only tried sin and
>> log, gave me the same performance (on a Intel processor!) like Intels
>> VML .
>>
>> I was also playing around with the block size in numexpr. What are
>> the rationale that led to the current block size of 128? Especially
>> with VML, a larger block size of 4096 instead of 128 allowed  to
>> efficiently use multithreading in VML.
>>     
>
> Experimentation.  Back in 2006 David found that 128 was optimal for the 
> processors available by that time.  With the Numexpr 1.1 my experiments 
> show that 256 is a better value for current Core2 processors and most 
> expressions in our benchmark bed (see benchmarks/ directory); hence, 
> 256 is the new value for the chunksize in 1.1.  However, be in mind 
> that 256 has to be multiplied by the itemsize of each array, so the 
> chunksize is currently 2048 bytes for 64-bit items (int64 or float64) 
> and 4096 for double precision complex arrays, which are probably the 
> sizes that have to be compared with VML.
>   
So the optimum block size  might depend  on the type of expression and 
if VML functions are used. On question: the block size is set by a 
#define, is there a significantly poorer performance if you use a 
variable instead? Would be more flexible, especially for testing and tuning.
>> I was missing the support for single precision floats.
>>     
>
> Yeah.  This is because nobody has implemented it before, but it is 
> completely doable.
>   



From bolme1234 at comcast.net  Fri Jan 16 15:20:37 2009
From: bolme1234 at comcast.net (David Bolme)
Date: Fri, 16 Jan 2009 13:20:37 -0700
Subject: [Numpy-discussion] Help with interpolating missing values from
	a 3D scanner
In-Reply-To: <6a17e9ee0901152342l533ec36cp24ac8918be1e3bf3@mail.gmail.com>
References: <gknb08$pnq$1@ger.gmane.org>
	<EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
	<3d375d730901151614h2c3b8478y9900bba97dbd8381@mail.gmail.com>
	<6a17e9ee0901152342l533ec36cp24ac8918be1e3bf3@mail.gmail.com>
Message-ID: <6D8BCD33-1CC6-4373-96E8-F1D054C32548@comcast.net>



>

Thanks for all the ideas.  I think I will look into the  
scikits.delaunay, Rbf, or gaussian smoothing approach.  My best idea  
is similar to the Gaussian smoothing.  Anyway, all of the missing data  
gaps seem to be small enough that I expect any of these methods to  
accomplish my purpose.   I have read some of the work on PDE's and in- 
painting but I think it is overkill for this particular application.    
I will let you know how it works.

Thanks again,
Dave


From matthew.brett at gmail.com  Fri Jan 16 19:07:30 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 16 Jan 2009 19:07:30 -0500
Subject: [Numpy-discussion] Please don't use google code for hosting
Message-ID: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>

Hi,

I am just visiting colleagues in the Cuban Neuroscience Center, and of
course I'm trying to persuade them that Python and open-source are the
way forward.

This is made more difficult because several projects - for example
pyglet - have their repositories on Google code.  Google, unlike any
other hoster I've come across, bans code downloads from Cuban
addresses with a message that I am trying to download from 'a
forbidden country'.  I assume they are playing safe with code export
regulations.

This breaks easy_install and direct downloads.

I hope you agree with me that it serves nobody's interest to reduce
collaboration with Cuban scientists on free software.

So, please, if you are considering google code for hosting, consider
other options.

Best,

Matthew


From bioinformed at gmail.com  Fri Jan 16 19:24:56 2009
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Fri, 16 Jan 2009 19:24:56 -0500
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
Message-ID: <2e1434c10901161624n3aeef0e0y489e4e13938a749e@mail.gmail.com>

On Fri, Jan 16, 2009 at 7:07 PM, Matthew Brett <matthew.brett at gmail.com>wrote:

> So, please, if you are considering google code for hosting, consider
> other options.
>

Seems odd that you'd post that from a gmail account.  I do sympathize with
your suggestion, but I don't have a better alternative to Google code for my
project.  Maybe it would be best to address this limitation with Google
directly.

-Kevin Jacobs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090116/5298d592/attachment.html>

From alan at ajackson.org  Fri Jan 16 20:17:57 2009
From: alan at ajackson.org (Alan Jackson)
Date: Fri, 16 Jan 2009 19:17:57 -0600
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <2e1434c10901161624n3aeef0e0y489e4e13938a749e@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<2e1434c10901161624n3aeef0e0y489e4e13938a749e@mail.gmail.com>
Message-ID: <20090116191757.3cb22ed7@ajackson.org>

On Fri, 16 Jan 2009 19:24:56 -0500
"Kevin Jacobs <jacobs at bioinformed.com>" <bioinformed at gmail.com> wrote:

> On Fri, Jan 16, 2009 at 7:07 PM, Matthew Brett <matthew.brett at gmail.com>wrote:
> 
> > So, please, if you are considering google code for hosting, consider
> > other options.
> >
> 
> Seems odd that you'd post that from a gmail account.  I do sympathize with
> your suggestion, but I don't have a better alternative to Google code for my
> project.  Maybe it would be best to address this limitation with Google
> directly.

I have to worry a bit about export control laws at work, and I don't remember
what Cuba's status is, but I do know that for a small number of countries,
North Korea, Iran, and maybe Cuba, exporting anything of value is a potentially
serious crime with prison time - so I can fully appreciate why Google would
be careful. The US does prosecute, and they have levied amazingly large fines
against companies in recent years.

Quite possible those restrictions for Cuba will be loosened in the next few
months, from what I have read recently.

-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From robert.kern at gmail.com  Fri Jan 16 20:27:33 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 16 Jan 2009 19:27:33 -0600
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
Message-ID: <3d375d730901161727q6a90b54i9942492abd719c29@mail.gmail.com>

On Fri, Jan 16, 2009 at 18:07, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> I am just visiting colleagues in the Cuban Neuroscience Center, and of
> course I'm trying to persuade them that Python and open-source are the
> way forward.
>
> This is made more difficult because several projects - for example
> pyglet - have their repositories on Google code.  Google, unlike any
> other hoster I've come across, bans code downloads from Cuban
> addresses with a message that I am trying to download from 'a
> forbidden country'.  I assume they are playing safe with code export
> regulations.
>
> This breaks easy_install and direct downloads.
>
> I hope you agree with me that it serves nobody's interest to reduce
> collaboration with Cuban scientists on free software.
>
> So, please, if you are considering google code for hosting, consider
> other options.

As a workaround, you can ask the authors of the code your colleagues
are interested in to place their release tarballs on pypi in addition
to Google Code (the caveat being the 10MB/file limit imposed by the
admins. Complain to them, too!). For SVN access, you can probably set
up a bzr mirror on launchpad for them.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From simpson at math.toronto.edu  Fri Jan 16 22:14:16 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Fri, 16 Jan 2009 22:14:16 -0500
Subject: [Numpy-discussion] tensor contractions
Message-ID: <058F21EC-8B7B-4F68-8772-0AE48151FFD0@math.toronto.edu>

Suppose I have a 3d array, A, with dimensions 2 x 2 x N, and a 2d  2 x  
N array, u.  I interpret A as N 2x2 matrices and u as N 2d vectors.   
Suppose I want to apply the mth matrix to the mth vector, i.e.

A[, , m] u[, m] = v[, m]

Aside from doing
A[0,0,:] u[0,:]  + A[0,1,:] u[1,:] = v[0,:]
and
A[1,0,:] u[0,:]  + A[1,1,:] u[1,:] = v[1,:]
is there a smart way to perform this computation?

-gideon



From ted.horst at earthlink.net  Fri Jan 16 23:54:42 2009
From: ted.horst at earthlink.net (Ted Horst)
Date: Fri, 16 Jan 2009 22:54:42 -0600
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161748.34558.faltet@pytables.org>
References: <200901161300.26130.faltet@pytables.org>
	<49709C3D.1010608@googlemail.com>
	<200901161748.34558.faltet@pytables.org>
Message-ID: <76014571-3BFF-44B4-BB8A-04DC4B5F513C@earthlink.net>

Note that Apple has a similar library called vForce:

<http://developer.apple.com/ReleaseNotes/Performance/RN-vecLib/index.html 
 >
<http://developer.apple.com/documentation/Performance/Conceptual/vecLib/Reference/reference.html 
 >

I think these libraries use several techniques and are not necessarily  
dependent on SSE.  The apple versions appear to only support float and  
double (no complex), and I don't see anything about strided arrays.   
At one point I thought there was talk of adding support for vForce  
into the respective ufuncs.  I don't know if anybody followed up on  
that.

On 2009-01-16, at 10:48, Francesc Alted wrote:

> Wow, pretty nice speed-ups indeed!  In fact I was thinking in  
> including
> support for threading in Numexpr (I don't think it would be too
> difficult, but let's see).  BTW, do you know how VML is able to  
> achieve
> a speedup of 6x for a sin() function?  I suppose this is because they
> are using SSE instructions, but, are these also available for 64-bit
> double precision items?



From olivier.grisel at ensta.org  Sat Jan 17 01:37:55 2009
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Sat, 17 Jan 2009 07:37:55 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <4970E189.8000906@googlemail.com>
References: <200901161300.26130.faltet@pytables.org>
	<49709C3D.1010608@googlemail.com>
	<200901161748.34558.faltet@pytables.org>
	<4970E189.8000906@googlemail.com>
Message-ID: <db0bec8f0901162237k222eeb93x73e3dac7d3a4d080@mail.gmail.com>

2009/1/16 Gregor Thalhammer <gregor.thalhammer at gmail.com>:
> Francesc Alted schrieb:
>>
>> Wow, pretty nice speed-ups indeed!  In fact I was thinking in including
>> support for threading in Numexpr (I don't think it would be too
>> difficult, but let's see).  BTW, do you know how VML is able to achieve
>> a speedup of 6x for a sin() function?  I suppose this is because they
>> are using SSE instructions, but, are these also available for 64-bit
>> double precision items?
>>
> I am not an expert on SSE instructions, but to my knowledge there exist
> (in the Core 2 architecture) no SSE instruction to calculate the sin.
> But it seems to be possible to (approximately) calculate a sin with a
> couple of multiplication/ addition instructions (and they exist in SSE
> for 64-bit float). Intel (and AMD) seems to use a more clever algorithm,

Here is the lib I use for the  transcendental functions SSE implementations:
http://gruntthepeon.free.fr/ssemath/

(only simple precision float though).

-- 
Olivier


From cournape at gmail.com  Sat Jan 17 04:50:15 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 17 Jan 2009 18:50:15 +0900
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <4970E189.8000906@googlemail.com>
References: <200901161300.26130.faltet@pytables.org>
	<49709C3D.1010608@googlemail.com>
	<200901161748.34558.faltet@pytables.org>
	<4970E189.8000906@googlemail.com>
Message-ID: <5b8d13220901170150g448682ebhd5615a5080ecd04@mail.gmail.com>

On Sat, Jan 17, 2009 at 4:35 AM, Gregor Thalhammer
<gregor.thalhammer at gmail.com> wrote:
> Francesc Alted schrieb:
>> A Friday 16 January 2009, Gregor Thalhammer escrigu?:
>>
>>> I also gave a try to the vector math library (VML), contained in
>>> Intel's Math Kernel Library. This offers a fast implementation of
>>> mathematical functions, operating on array. First I implemented a C
>>> extension, providing new ufuncs. This gave me a big performance gain,
>>> e.g., 2.3x (5x) for sin, 6x (10x) for exp, 7x (15x) for pow, and 3x
>>> (6x) for division (no gain for add, sub, mul).
>>>
>>
>> Wow, pretty nice speed-ups indeed!  In fact I was thinking in including
>> support for threading in Numexpr (I don't think it would be too
>> difficult, but let's see).  BTW, do you know how VML is able to achieve
>> a speedup of 6x for a sin() function?  I suppose this is because they
>> are using SSE instructions, but, are these also available for 64-bit
>> double precision items?
>>
> I am not an expert on SSE instructions, but to my knowledge there exist
> (in the Core 2 architecture) no SSE instruction to calculate the sin.
> But it seems to be possible to (approximately) calculate a sin with a
> couple of multiplication/ addition instructions (and they exist in SSE
> for 64-bit float). Intel (and AMD) seems to use a more clever algorithm,
> efficiently implemented than the standard implementation.

Generally, transcendent functions are not sped up because they are
implemented in hardware. There is no special algorithm: you implement
those as you would in C using Taylor expansions or other known
polynomial expansions, except you use SIMD to implement those
polynomial expansions. You can also use table lookup, which can be
pretty fast to get full precision for trigonometric functions.
musicdsp.org has some of those (take care a lot of those tricks do not
give full precision - it is used for music synthesis, where full
precision is rarely needed and speed of uttermost importance):

http://www.musicdsp.org

There were some examples on freescale.com, in full precision, but I
can't find it anymore.

For some functions, you can get almost one order of magnitude faster
transcendental functions (for full precision), but it is a lot of work
to make sure they work as expected in a cross platform way (even
limiting to one CPU arch when using asm, there are differences between
compilers which make this rather difficult).


From grh at mur.at  Sat Jan 17 05:05:11 2009
From: grh at mur.at (Georg Holzmann)
Date: Sat, 17 Jan 2009 11:05:11 +0100
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
Message-ID: <4971AD57.2000605@mur.at>

Hallo!

Thanks, this is interesting !

Do you also know how the situation is with sourceforge/launchpad/trac... 
and other popular hosting systems ?
Do they also have these restrictions ?

LG
Georg


Matthew Brett schrieb:
> Hi,
> 
> I am just visiting colleagues in the Cuban Neuroscience Center, and of
> course I'm trying to persuade them that Python and open-source are the
> way forward.
> 
> This is made more difficult because several projects - for example
> pyglet - have their repositories on Google code.  Google, unlike any
> other hoster I've come across, bans code downloads from Cuban
> addresses with a message that I am trying to download from 'a
> forbidden country'.  I assume they are playing safe with code export
> regulations.
> 
> This breaks easy_install and direct downloads.
> 
> I hope you agree with me that it serves nobody's interest to reduce
> collaboration with Cuban scientists on free software.
> 
> So, please, if you are considering google code for hosting, consider
> other options.
> 
> Best,
> 
> Matthew
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From dsdale24 at gmail.com  Sat Jan 17 23:06:11 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sat, 17 Jan 2009 23:06:11 -0500
Subject: [Numpy-discussion] question about memory ownership
Message-ID: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>

Hello,

Would someone be so kind as to explain how to create an ndarray subclass
that owns its own memory? I think RealisticInfoArray at
http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#basics-subclassingdoes
not own its own memory, do you have to call ndarray.__new__ directly,
or is there another way?

Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090117/146ef195/attachment.html>

From robert.kern at gmail.com  Sat Jan 17 23:23:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 17 Jan 2009 22:23:38 -0600
Subject: [Numpy-discussion] question about memory ownership
In-Reply-To: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>
References: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>
Message-ID: <3d375d730901172023g18ea9ea2wd13b78533c5a4b95@mail.gmail.com>

On Sat, Jan 17, 2009 at 22:06, Darren Dale <dsdale24 at gmail.com> wrote:
> Hello,
>
> Would someone be so kind as to explain how to create an ndarray subclass
> that owns its own memory? I think RealisticInfoArray at
> http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#basics-subclassing
> does not own its own memory, do you have to call ndarray.__new__ directly,
> or is there another way?

Does it matter? Unless if something actually digs down to grab the
base object, the RealisticInfoArray object is the only thing keeping
it alive. When it goes away, the underlying array goes away.

But take a look at the implementation of memmap for a subclass that
calls ndarray.__new__ directly.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dsdale24 at gmail.com  Sat Jan 17 23:35:54 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sat, 17 Jan 2009 23:35:54 -0500
Subject: [Numpy-discussion] question about memory ownership
In-Reply-To: <3d375d730901172023g18ea9ea2wd13b78533c5a4b95@mail.gmail.com>
References: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>
	<3d375d730901172023g18ea9ea2wd13b78533c5a4b95@mail.gmail.com>
Message-ID: <a08e5f80901172035g31d6aee9ue9ee52f288fb89d0@mail.gmail.com>

On Sat, Jan 17, 2009 at 11:23 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Jan 17, 2009 at 22:06, Darren Dale <dsdale24 at gmail.com> wrote:
> > Hello,
> >
> > Would someone be so kind as to explain how to create an ndarray subclass
> > that owns its own memory? I think RealisticInfoArray at
> >
> http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#basics-subclassing
> > does not own its own memory, do you have to call ndarray.__new__
> directly,
> > or is there another way?
>
> Does it matter? Unless if something actually digs down to grab the
> base object, the RealisticInfoArray object is the only thing keeping
> it alive. When it goes away, the underlying array goes away.
>
> But take a look at the implementation of memmap for a subclass that
> calls ndarray.__new__ directly.
>

I'm still working on my physical quantities package, and I want to prevent
something like the following:

q=[1,2,3,4]*kg*m**2/s**2
qq=q[:2]
qq.units=BTU

That would give me qq in units of british thermal units, but it would also
modify the magnitude of q[:2]. It seems the most straight forward solution
is to simply disallow in-place modification of units when the array doesn't
own its own memory.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090117/2c13fb5c/attachment.html>

From robert.kern at gmail.com  Sat Jan 17 23:44:05 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 17 Jan 2009 22:44:05 -0600
Subject: [Numpy-discussion] question about memory ownership
In-Reply-To: <a08e5f80901172035g31d6aee9ue9ee52f288fb89d0@mail.gmail.com>
References: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>
	<3d375d730901172023g18ea9ea2wd13b78533c5a4b95@mail.gmail.com>
	<a08e5f80901172035g31d6aee9ue9ee52f288fb89d0@mail.gmail.com>
Message-ID: <3d375d730901172044uf3fd4b2k612167ad2db7affe@mail.gmail.com>

On Sat, Jan 17, 2009 at 22:35, Darren Dale <dsdale24 at gmail.com> wrote:
>
> On Sat, Jan 17, 2009 at 11:23 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Sat, Jan 17, 2009 at 22:06, Darren Dale <dsdale24 at gmail.com> wrote:
>> > Hello,
>> >
>> > Would someone be so kind as to explain how to create an ndarray subclass
>> > that owns its own memory? I think RealisticInfoArray at
>> >
>> > http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#basics-subclassing
>> > does not own its own memory, do you have to call ndarray.__new__
>> > directly,
>> > or is there another way?
>>
>> Does it matter? Unless if something actually digs down to grab the
>> base object, the RealisticInfoArray object is the only thing keeping
>> it alive. When it goes away, the underlying array goes away.
>>
>> But take a look at the implementation of memmap for a subclass that
>> calls ndarray.__new__ directly.
>
>
> I'm still working on my physical quantities package, and I want to prevent
> something like the following:
>
> q=[1,2,3,4]*kg*m**2/s**2
> qq=q[:2]
> qq.units=BTU
>
> That would give me qq in units of british thermal units, but it would also
> modify the magnitude of q[:2]. It seems the most straight forward solution
> is to simply disallow in-place modification of units when the array doesn't
> own its own memory.

Ah. I'd take a functional approach to unit conversion, then. You can't
modify units in-place. Memory can't be much of a concern since the
conversion code will have temporaries.

But you could check that .base is an instance of your quantity class.
If it's a regular array, go ahead and modify in-place.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From simpson at math.toronto.edu  Sun Jan 18 00:30:21 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sun, 18 Jan 2009 00:30:21 -0500
Subject: [Numpy-discussion] efficient usage of tensordot
Message-ID: <9A822472-3C2D-4EE3-88E4-D2C8528868E1@math.toronto.edu>

This is related to a question I posted earlier.

Suppose I have array A with dimensions n x m x l and array x with  
dimensions m x l.  Interpret this as an array of l nxm matrices and  
and array of l m dimensional vectors.  I wish to compute the matrix- 
vector product A[:,:,k] x[:,k] for each k = 0,... l -1.  I discovered  
that I could accomplish this with the command

np.diagonal(np.tensordot(A, k, axes=(1,0)), axis1= 1, axis2 = 2)

The tensordot command gives me

A_{ijk}x_{jl} = C_{ikl}

And the diagonal command grabs the entries in array C where k=l.

Is this the "optimal" way to make this calculation in numpy?  It  
certainly makes for nice, clean code, but is it the fastest I can get?


-gideon



From nadavh at visionsense.com  Sun Jan 18 03:38:31 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sun, 18 Jan 2009 10:38:31 +0200
Subject: [Numpy-discussion] efficient usage of tensordot
References: <9A822472-3C2D-4EE3-88E4-D2C8528868E1@math.toronto.edu>
Message-ID: <710F2847B0018641891D9A216027636029C3BD@ex3.envision.co.il>

This is not the first time this issue is raised here.
You may try this piece of code, which may take less memory:

(A*x).sum(axis=1).T

  Nadav



-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Gideon Simpson
????: ? 18-?????-09 07:30
??: Discussion of Numerical Python
????: [Numpy-discussion] efficient usage of tensordot
 
This is related to a question I posted earlier.

Suppose I have array A with dimensions n x m x l and array x with  
dimensions m x l.  Interpret this as an array of l nxm matrices and  
and array of l m dimensional vectors.  I wish to compute the matrix- 
vector product A[:,:,k] x[:,k] for each k = 0,... l -1.  I discovered  
that I could accomplish this with the command

np.diagonal(np.tensordot(A, k, axes=(1,0)), axis1= 1, axis2 = 2)

The tensordot command gives me

A_{ijk}x_{jl} = C_{ikl}

And the diagonal command grabs the entries in array C where k=l.

Is this the "optimal" way to make this calculation in numpy?  It  
certainly makes for nice, clean code, but is it the fastest I can get?


-gideon

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3412 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090118/9997a328/attachment.bin>

From dsdale24 at gmail.com  Sun Jan 18 08:17:53 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 18 Jan 2009 08:17:53 -0500
Subject: [Numpy-discussion] question about memory ownership
In-Reply-To: <3d375d730901172044uf3fd4b2k612167ad2db7affe@mail.gmail.com>
References: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>
	<3d375d730901172023g18ea9ea2wd13b78533c5a4b95@mail.gmail.com>
	<a08e5f80901172035g31d6aee9ue9ee52f288fb89d0@mail.gmail.com>
	<3d375d730901172044uf3fd4b2k612167ad2db7affe@mail.gmail.com>
Message-ID: <a08e5f80901180517r36c7f251oba9ade91c4152036@mail.gmail.com>

On Sat, Jan 17, 2009 at 11:44 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Jan 17, 2009 at 22:35, Darren Dale <dsdale24 at gmail.com> wrote:
> >
> > On Sat, Jan 17, 2009 at 11:23 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Sat, Jan 17, 2009 at 22:06, Darren Dale <dsdale24 at gmail.com> wrote:
> >> > Hello,
> >> >
> >> > Would someone be so kind as to explain how to create an ndarray
> subclass
> >> > that owns its own memory? I think RealisticInfoArray at
> >> >
> >> >
> http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#basics-subclassing
> >> > does not own its own memory, do you have to call ndarray.__new__
> >> > directly,
> >> > or is there another way?
> >>
> >> Does it matter? Unless if something actually digs down to grab the
> >> base object, the RealisticInfoArray object is the only thing keeping
> >> it alive. When it goes away, the underlying array goes away.
> >>
> >> But take a look at the implementation of memmap for a subclass that
> >> calls ndarray.__new__ directly.
> >
> >
> > I'm still working on my physical quantities package, and I want to
> prevent
> > something like the following:
> >
> > q=[1,2,3,4]*kg*m**2/s**2
> > qq=q[:2]
> > qq.units=BTU
> >
> > That would give me qq in units of british thermal units, but it would
> also
> > modify the magnitude of q[:2]. It seems the most straight forward
> solution
> > is to simply disallow in-place modification of units when the array
> doesn't
> > own its own memory.
>
> Ah. I'd take a functional approach to unit conversion, then. You can't
> modify units in-place. Memory can't be much of a concern since the
> conversion code will have temporaries.
>
> But you could check that .base is an instance of your quantity class.
> If it's a regular array, go ahead and modify in-place.
>

Thanks for the advice. I'd like to look into the second option, since the
former would not allow other in-place operations like *= /= **/. Hopefully
the package will be fit to share with this list before too long.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090118/fa83b22e/attachment.html>

From bolme1234 at comcast.net  Sun Jan 18 13:31:25 2009
From: bolme1234 at comcast.net (David Bolme)
Date: Sun, 18 Jan 2009 11:31:25 -0700
Subject: [Numpy-discussion] Help with interpolating missing values from
	a 3D scanner
In-Reply-To: <EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
References: <gknb08$pnq$1@ger.gmane.org>
	<EB00E9BF-BF0C-4D4C-A1BF-BE36B424F9F1@comcast.net>
Message-ID: <D0AB6283-5BB1-4549-B4A4-E72E70A57A51@comcast.net>

I have implemented an iterative gaussian smoothing approach that is  
working well for my purposes.  My approach uses a median filter to  
populate the initial values and then runs a few passes with gaussian  
smoothing.   This works very well for the missing values that I care  
about within the face region.

I also came across an error when I tried to use the Rbf class.  I was  
hoping that I could just input all of the data that I have and have a  
quick and easy solution.  I expect this would work if ran the Rbf on  
just a small image tile near the missing data region.  I am not sure  
if this is worthy of a bug report.

When I tried to create an RBF from the full image I got this error:

Traceback (most recent call last):
  File "/Users/bolme/Documents/workspace/pyvision/src/pyvision/types/ 
RangeImage.py", line 258, in <module>
    ri.populateMissingData()
  File "/Users/bolme/Documents/workspace/pyvision/src/pyvision/types/ 
RangeImage.py", line 184, in populateMissingData
    it.Rbf(x[mask],y[mask],z[mask])
  File "/Library/Python/2.5/site-packages/scipy-0.7.0.dev4645-py2.5- 
macosx-10.3-i386.egg/scipy/interpolate/rbf.py", line 129, in __init__
    r = self._call_norm(self.xi, self.xi)
  File "/Library/Python/2.5/site-packages/scipy-0.7.0.dev4645-py2.5- 
macosx-10.3-i386.egg/scipy/interpolate/rbf.py", line 144, in _call_norm
    return self.norm(x1, x2)
  File "/Library/Python/2.5/site-packages/scipy-0.7.0.dev4645-py2.5- 
macosx-10.3-i386.egg/scipy/interpolate/rbf.py", line 54, in  
_euclidean_norm
    return sqrt( ((x1 - x2)**2).sum(axis=0) )
ValueError: broadcast dimensions too large.

This is probably because I tried to input the full 640X480 image.  Too  
much data.  x[mask], y[mask], and z[mask] are a one dimensional arrays  
with approximately 100,000 elements.  I am trying to predict z.  It  
would be nice to have a more descriptive error message.



From jh at physics.ucf.edu  Sun Jan 18 16:21:42 2009
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Sun, 18 Jan 2009 16:21:42 -0500
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161704.51777.faltet@pytables.org> (message from Francesc
	Alted on Fri, 16 Jan 2009 17:04:51 +0100)
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
	<200901161704.51777.faltet@pytables.org>
Message-ID: <wl8tz7wfgyh.fsf@glup.physics.ucf.edu>

Francesc Alted wrote:

> > > Numexpr is a fast numerical expression evaluator for NumPy.  With
> > > it, expressions that operate on arrays (like "3*a+4*b") are
> > > accelerated and use less memory than doing the same calculation in
> > > Python.
>
> > Please pardon my ignorance as I know this project has been around for
> > a while.  It this looks very exciting, but either it's cumbersome, or
> > I'm not understanding exactly what's being fixed.  If you can
> > accelerate evaluation, why not just integrate the faster math into
> > numpy, rather than having two packages?  Or is this something that is
> > only an advantage when the expression is given as a string (and why
> > is that the case)?  It would be helpful if you could put the answer
> > on your web page and in your standard release blurb in some compact
> > form. I guess what I'm really looking for when I read one of those is
> > a quick answer to the question "should I look into this?".

> Well, there is a link in the project page to the "Overview" section of 
> the wiki, but perhaps is a bit hidden.  I've added some blurb as you 
> suggested in the main page an another link to the "Overview" wiki page.
> Hope that, by reading the new blurb, you can see why it accelerates 
> expression evaluation with regard to NumPy.  If not, tell me and will 
> try to come with something more comprehensible.

I did see the overview.  The addition you made is great but it's so
far down that many won't get to it.  Even in its section, the meat of
it is below three paragraphs that most users won't care about and many
won't understand.  I've posted some notes on writing intros in
Developer_Zone.

In the following, I've reordered the page to address the questions of
potential users first, edited it a bit, and fixed the example to
conform to our doc standards (and 128->256; hope that was right).  See
what you think...

** Description:

The numexpr package evaluates multiple-operator array expressions many
times faster than numpy can.  It accepts the expression as a string,
analyzes it, rewrites it more efficiently, and compiles it to faster
Python code on the fly.  It's the next best thing to writing the
expression in C and compiling it with an optimizing compiler (as
scipy.weave does), but requires no compiler at runtime.

Using it is simple:

>>> import numpy as np
>>> import numexpr as ne
>>> a = np.arange(10)
>>> b = np.arange(0, 20, 2)
>>> c = ne.evaluate("2*a+3*b")
>>> c
array([ 0,  8, 16, 24, 32, 40, 48, 56, 64, 72])

** Why does it work?

There are two extremes to array expression evaluation.  Each binary
operation can run separately over the array elements and return a
temporary array.  This is what NumPy does: 2*a + 3*b uses three
temporary arrays as large as a or b.  This strategy wastes memory (a
problem if the arrays are large).  It is also not a good use of CPU
cache memory because the results of 2*a and 3*b will not be in cache
for the final addition if the arrays are large.

The other extreme is to loop over each element:

for i in xrange(len(a)):
    c[i] = 2*a[i] + 3*b[i]

This conserves memory and is good for the cache, but on each iteration
Python must check the type of each operand and select the correct
routine for each operation.  All but the first such checks are wasted,
as the input arrays are not changing.

numexpr uses an in-between approach.  Arrays are handled in chunks
(the first pass uses 256 elements).  As Python code, it looks
something like this:

for i in xrange(0, len(a), 256):
    r0 = a[i:i+256]
    r1 = b[i:i+256]
    multiply(r0, 2, r2)
    multiply(r1, 3, r3)
    add(r2, r3, r2)
    c[i:i+256] = r2

The 3-argument form of add() stores the result in the third argument,
instead of allocating a new array.  This achieves a good balance
between cache and branch prediction.  The virtual machine is written
entirely in C, which makes it faster than the Python above.

** Supported Operators (unchanged)

** Supported Functions (unchanged, but capitalize 'F')

** Usage Notes (no need to repeat the example)

Numexpr's principal routine is:

evaluate(ex, local_dict=None, global_dict=None, **kwargs)

ex is a string forming an expression, like "2*a+3*b".  The values for
a and b will by default be taken from the calling function's frame
(through the use of sys._getframe()).  Alternatively, they can be
specified using the local_dict or global_dict` arguments, or passed as
keyword arguments.

Expressions are cached, so reuse is fast.  Arrays or scalars are
allowed for the variables, which must be of type 8-bit boolean (bool),
32-bit signed integer (int), 64-bit signed integer (long),
double-precision floating point number (float), 2x64-bit,
double-precision complex number (complex) or raw string of bytes
(str).  The arrays must all be the same size.

** Building (unchanged, but move down since it's standard and most
   users will only do this once, if ever)

** Implementation Notes (rest of current How It Works section)

** Credits

--jh--


From robert.kern at gmail.com  Mon Jan 19 04:27:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 19 Jan 2009 03:27:02 -0600
Subject: [Numpy-discussion] question about memory ownership
In-Reply-To: <a08e5f80901180517r36c7f251oba9ade91c4152036@mail.gmail.com>
References: <a08e5f80901172006s7d4d5abcxa882646e7ba260b5@mail.gmail.com>
	<3d375d730901172023g18ea9ea2wd13b78533c5a4b95@mail.gmail.com>
	<a08e5f80901172035g31d6aee9ue9ee52f288fb89d0@mail.gmail.com>
	<3d375d730901172044uf3fd4b2k612167ad2db7affe@mail.gmail.com>
	<a08e5f80901180517r36c7f251oba9ade91c4152036@mail.gmail.com>
Message-ID: <3d375d730901190127med2cf85x350efe72111c6d82@mail.gmail.com>

On Sun, Jan 18, 2009 at 07:17, Darren Dale <dsdale24 at gmail.com> wrote:

> Thanks for the advice. I'd like to look into the second option, since the
> former would not allow other in-place operations like *= /= **/.

That is a good use case.

> Hopefully
> the package will be fit to share with this list before too long.

Great!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From yakov.keselman at gmail.com  Mon Jan 19 04:49:12 2009
From: yakov.keselman at gmail.com (Yakov Keselman)
Date: Mon, 19 Jan 2009 01:49:12 -0800
Subject: [Numpy-discussion] How to make "lazy" derived arrays in a
	recarray view of a memmap on large files
In-Reply-To: <f2e7a4240901160302w758afc31oe39a9952ca53c975@mail.gmail.com>
References: <f2e7a4240901160302w758afc31oe39a9952ca53c975@mail.gmail.com>
Message-ID: <dc3116ef0901190149m1a474971k7d8e330bf3022802@mail.gmail.com>

The problem, as I understand it, is this:

you have a large array and you want to define objects that (1) behave
like arrays; (2) are derived from the large array (can be computed
from it); (3) should not take much space if only small portions of the
large array are ever referenced.

A simple solution that would satisfy (2) and (3) would be to define a
method that uses indices and returns real arrays, so that instead of
saying "besp.bacon[1000000:1001000]" you'd say
"besp.bacon(1000000:1001000)". You'd do >>4 inside the method only on
the portion of the original array that you're interested in.

A solution that would also satisfy (1) would need to implement
array-like methods on your newly-defined class. You can probably use
http://docs.python.org/reference/datamodel.html#emulating-container-types
as your starting point.

If your code is meant only for yourself, I'd suggest to go with the
inconvenient but working method approach. If other people will want to
use your class in an array-like manner, you'd have to properly define
all the functionality that people would expect from an array.

Hope this helps.

= Yakov


On 1/16/09, Kim Hansen <slaunger at gmail.com> wrote:
> Hi numpy forum
>
> I need to efficiently handle some large (300 MB) recordlike binary
> files, where some data fields are less than a byte and thus cannot be
> mapped in a record dtype immediately.
>
> I would like to be able to access these derived arrays in a memory
> efficient manner but I cannot figure out how to acheive this.
>
> My application of the derived arrays would never be to do operation on
> the entire array, rather iterate over some selected elements and do
> somthing about it - operations which seems well suited for doing on
> demand
>
> I wrote a related post yesterday, which I have not received any
> response on. I am now posting again using another example and perhaps
> more clear example which I beleive describes my problem spot on
>
> from numpy import *
>
> # Python.exe memory use here: 8.14 MB
> desc = dtype([("baconandeggs", "<u1"), ("spam","<u1"), ("parrots","<u1")])
> index = memmap("g:/id-2008-10-25-17-ver4.idx", dtype = desc,
> mode="r").view(recarray)
> # The index file is very large, contains 292 MB of data
> # Python.exe memory use: 8.16 MB, only 20 kB extra for memmap mapped to
> recarray
>
> # The following instant operation takes a few secs working on 3*10^8
> elements
> # How can I derive new array in a lazy/ondemand/memmap manner?
> index.bacon = index.baconandeggs >> 4
> # python.exe memory use: 595 MB! Not surprising but how to do better??
>
> # Another derived array, which is resource demanding
> index.eggs = index.baconandeggs & 0x0F
> # python.exe memory usage is now 731 MB!
>
> What I'd like to do is implement a class, LazyBaconEggsSpamParrots,
> which encapsulates the
> derived arrays
>
> such that I could do
>
> besp = LazyBaconEggsSpamParrots("baconeggsspamparrots.idx")
> for b in besp.bacon: #Iterate lazy
>     spam(b)
> #Only derive the 1000 needed elements, don't do all 1000000
> dosomething(besp.bacon[1000000:1001000])
>
> I envision the class would look something like this
>
> class LazyBaconEggsSpamParrots(object):
>
>     def __init__(self, filename):
>          desc = dtype([("baconandeggs", "<u1"),
>                        ("spam","<u1"),
>                        ("parrots","<u1")])
>          self._data = memmap(filename, dtype=desc, mode='r').view(recarray)
>          # Expose the one-to-one data directly
>          self.spam = self._data.spam
>          self.parrots = self._data.parrots
>          # This would work but costs way too much memory
>          # self.bacon = self._data.baconandeggs >> 4
>          # self.eggs = self._data.baconandeggs & 0x0F
>
>     def __getattr__(self, attr_name):
>         if attr_name == "bacon":
>             # return bacon in an on demand manner, but how?
>         elif attr_name == "eggs":
>             # return eggs in an on demand manner, but how?
>         else:
>             # If the name is not a data attribute treat it as a normal
>             # non-existing attribute - raise AttributeError
>             raise AttributeError
>
> but how to do the lazy part of it?
>
> -- Kim
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


-- 
Not to laugh, not to lament, not to curse, but to understand. -- Spinoza


From slaunger at gmail.com  Mon Jan 19 05:23:01 2009
From: slaunger at gmail.com (Kim Hansen)
Date: Mon, 19 Jan 2009 11:23:01 +0100
Subject: [Numpy-discussion] How to make "lazy" derived arrays in a
	recarray view of a memmap on large files
In-Reply-To: <dc3116ef0901190149m1a474971k7d8e330bf3022802@mail.gmail.com>
References: <f2e7a4240901160302w758afc31oe39a9952ca53c975@mail.gmail.com>
	<dc3116ef0901190149m1a474971k7d8e330bf3022802@mail.gmail.com>
Message-ID: <f2e7a4240901190223h12b4efd8t6c0b03ad3aa07c9d@mail.gmail.com>

Hi Yakov,

Thank you for you kind advice. I ended up doing something simpler and
less arcane.

I first read the baconandeggs interleaved data from a memmap, create a
new writeable memmap based on an unpacked dtype, where bacon and eggs
are in two different '<u1' fields, and then I simply convert the
interleaved baconandeggs data once and for all into the unpacked
(bacon, eggs) data structure in another file. If I do the array
conversions as one-lines I get transient large memory use as expected,
but the memory usage goes down again after taking the few seconds it
takes to convert the data once and for all. And if I have really large
arrays, I have the option of doing the copying in chunks of say,
200000 elements using slices. In that way I end up having a standard
memmapped recarray, where all fields are easily accessible in a
transparent manner, without having to implement and maintain the
container-type logic. So I have transformed it into a simple data
transformation problem, where the conversion code is centralized and
easily maintained if I want to add a new field, etc.

Not rocket science, but it works. It also has the advantage, that
subsequent access to the unpacked data is efficient and
straight-forward, which is good for my application, as i need to use
the data afterwards in many different ways for many different
applications.

Cheers,

Kim



2009/1/19 Yakov Keselman <yakov.keselman at gmail.com>:
> The problem, as I understand it, is this:
>
> you have a large array and you want to define objects that (1) behave
> like arrays; (2) are derived from the large array (can be computed
> from it); (3) should not take much space if only small portions of the
> large array are ever referenced.
>
> A simple solution that would satisfy (2) and (3) would be to define a
> method that uses indices and returns real arrays, so that instead of
> saying "besp.bacon[1000000:1001000]" you'd say
> "besp.bacon(1000000:1001000)". You'd do >>4 inside the method only on
> the portion of the original array that you're interested in.
>
> A solution that would also satisfy (1) would need to implement
> array-like methods on your newly-defined class. You can probably use
> http://docs.python.org/reference/datamodel.html#emulating-container-types
> as your starting point.
>
> If your code is meant only for yourself, I'd suggest to go with the
> inconvenient but working method approach. If other people will want to
> use your class in an array-like manner, you'd have to properly define
> all the functionality that people would expect from an array.
>
> Hope this helps.
>
> = Yakov
>
>
> On 1/16/09, Kim Hansen <slaunger at gmail.com> wrote:
>> Hi numpy forum
>>
>> I need to efficiently handle some large (300 MB) recordlike binary
>> files, where some data fields are less than a byte and thus cannot be
>> mapped in a record dtype immediately.
>>
>> I would like to be able to access these derived arrays in a memory
>> efficient manner but I cannot figure out how to acheive this.
>>
>> My application of the derived arrays would never be to do operation on
>> the entire array, rather iterate over some selected elements and do
>> somthing about it - operations which seems well suited for doing on
>> demand
>>
>> I wrote a related post yesterday, which I have not received any
>> response on. I am now posting again using another example and perhaps
>> more clear example which I beleive describes my problem spot on
>>
>> from numpy import *
>>
>> # Python.exe memory use here: 8.14 MB
>> desc = dtype([("baconandeggs", "<u1"), ("spam","<u1"), ("parrots","<u1")])
>> index = memmap("g:/id-2008-10-25-17-ver4.idx", dtype = desc,
>> mode="r").view(recarray)
>> # The index file is very large, contains 292 MB of data
>> # Python.exe memory use: 8.16 MB, only 20 kB extra for memmap mapped to
>> recarray
>>
>> # The following instant operation takes a few secs working on 3*10^8
>> elements
>> # How can I derive new array in a lazy/ondemand/memmap manner?
>> index.bacon = index.baconandeggs >> 4
>> # python.exe memory use: 595 MB! Not surprising but how to do better??
>>
>> # Another derived array, which is resource demanding
>> index.eggs = index.baconandeggs & 0x0F
>> # python.exe memory usage is now 731 MB!
>>
>> What I'd like to do is implement a class, LazyBaconEggsSpamParrots,
>> which encapsulates the
>> derived arrays
>>
>> such that I could do
>>
>> besp = LazyBaconEggsSpamParrots("baconeggsspamparrots.idx")
>> for b in besp.bacon: #Iterate lazy
>>     spam(b)
>> #Only derive the 1000 needed elements, don't do all 1000000
>> dosomething(besp.bacon[1000000:1001000])
>>
>> I envision the class would look something like this
>>
>> class LazyBaconEggsSpamParrots(object):
>>
>>     def __init__(self, filename):
>>          desc = dtype([("baconandeggs", "<u1"),
>>                        ("spam","<u1"),
>>                        ("parrots","<u1")])
>>          self._data = memmap(filename, dtype=desc, mode='r').view(recarray)
>>          # Expose the one-to-one data directly
>>          self.spam = self._data.spam
>>          self.parrots = self._data.parrots
>>          # This would work but costs way too much memory
>>          # self.bacon = self._data.baconandeggs >> 4
>>          # self.eggs = self._data.baconandeggs & 0x0F
>>
>>     def __getattr__(self, attr_name):
>>         if attr_name == "bacon":
>>             # return bacon in an on demand manner, but how?
>>         elif attr_name == "eggs":
>>             # return eggs in an on demand manner, but how?
>>         else:
>>             # If the name is not a data attribute treat it as a normal
>>             # non-existing attribute - raise AttributeError
>>             raise AttributeError
>>
>> but how to do the lazy part of it?
>>
>> -- Kim
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> --
> Not to laugh, not to lament, not to curse, but to understand. -- Spinoza
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From meine at informatik.uni-hamburg.de  Mon Jan 19 05:34:47 2009
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Mon, 19 Jan 2009 11:34:47 +0100
Subject: [Numpy-discussion] new incremental statistics project
In-Reply-To: <ea7d6a710812181827t34572516j981df31fbadb3f07@mail.gmail.com>
References: <ea7d6a710812181827t34572516j981df31fbadb3f07@mail.gmail.com>
Message-ID: <200901191134.47779.meine@informatik.uni-hamburg.de>

On Friday 19 December 2008 03:27:12 Bradford Cross wrote:
> This is a new project I just released.
>
> I know it is C#, but some of the design and idioms would be nice in
> numpy/scipy for working with discrete event simulators, time series, and
> event stream processing.
>
> http://code.google.com/p/incremental-statistics/

Hi, do you know about the boost accumulators project?

It's still in boost's sandbox, but I love its design, and it provides a large 
number of well-documented, mathematically sound estimators for variance, mean, 
etc.:
http://boost-sandbox.sourceforge.net/libs/accumulators/doc/html/index.html

Just a heads-up, in case someone finds this useful here.
(Don't know about people's fondness of boost and/or C++ here.)

Greetings,
  Hans


From matthew.brett at gmail.com  Mon Jan 19 14:05:42 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 19 Jan 2009 14:05:42 -0500
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <2e1434c10901161624n3aeef0e0y489e4e13938a749e@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<2e1434c10901161624n3aeef0e0y489e4e13938a749e@mail.gmail.com>
Message-ID: <1e2af89e0901191105r3a2468c2r660d92030f1a059c@mail.gmail.com>

Hi,

> Seems odd that you'd post that from a gmail account.

I'm not saying Google in general is bad.   I'm just suggesting that,
in the particular case of Google code, it would allow greater openness
if you use something else.

> I do sympathize with
> your suggestion, but I don't have a better alternative to Google code for my
> project.

I sympathize too!

> Maybe it would be best to address this limitation with Google
> directly.

I'd like to change Google's policy, but I doubt my suggestion will
change their minds.  I'm most interested in making sure the code is
freely available (to everyone).

Best,

Matthew


From matthew.brett at gmail.com  Mon Jan 19 14:07:18 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 19 Jan 2009 14:07:18 -0500
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <3d375d730901161727q6a90b54i9942492abd719c29@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<3d375d730901161727q6a90b54i9942492abd719c29@mail.gmail.com>
Message-ID: <1e2af89e0901191107g1e4cbe2fvc03876fa105a00ce@mail.gmail.com>

Hi,

> As a workaround, you can ask the authors of the code your colleagues
> are interested in to place their release tarballs on pypi in addition
> to Google Code (the caveat being the 10MB/file limit imposed by the
> admins. Complain to them, too!). For SVN access, you can probably set
> up a bzr mirror on launchpad for them.

Thanks - very useful - and - it would be very good if those of us
hosting on google code could also post to pypi - solving most of the
current problem.

See you,

Matthew


From matthew.brett at gmail.com  Mon Jan 19 14:20:30 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 19 Jan 2009 14:20:30 -0500
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <4971AD57.2000605@mur.at>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<4971AD57.2000605@mur.at>
Message-ID: <1e2af89e0901191120k6b2880betb7e43c2607c1336d@mail.gmail.com>

Hi,

> Do you also know how the situation is with sourceforge/launchpad/trac...
> and other popular hosting systems ?
> Do they also have these restrictions ?

I've not noticed any problems with sourceforge, nor launchpad - I'm
using them regularly from here.   You'd hope that was the case for
launchpad - when my browser started this morning, it reminded me of
the Ubuntu / Canonical mission - http://www.canonical.com/aboutus

    1)  delivering the world's best free software platform
    2)  ensuring its availability to everyone
   ...

Best,

Matthew


From faltet at pytables.org  Mon Jan 19 15:04:21 2009
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 19 Jan 2009 21:04:21 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <wl8tz7wfgyh.fsf@glup.physics.ucf.edu>
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
	<200901161704.51777.faltet@pytables.org>
	<wl8tz7wfgyh.fsf@glup.physics.ucf.edu>
Message-ID: <200901192104.22227.faltet@pytables.org>

A Sunday 18 January 2009, jh at physics.ucf.edu escrigu?:
> Francesc Alted wrote:
> > > > Numexpr is a fast numerical expression evaluator for NumPy. 
> > > > With it, expressions that operate on arrays (like "3*a+4*b")
> > > > are accelerated and use less memory than doing the same
> > > > calculation in Python.
> > >
> > > Please pardon my ignorance as I know this project has been around
> > > for a while.  It this looks very exciting, but either it's
> > > cumbersome, or I'm not understanding exactly what's being fixed. 
> > > If you can accelerate evaluation, why not just integrate the
> > > faster math into numpy, rather than having two packages?  Or is
> > > this something that is only an advantage when the expression is
> > > given as a string (and why is that the case)?  It would be
> > > helpful if you could put the answer on your web page and in your
> > > standard release blurb in some compact form. I guess what I'm
> > > really looking for when I read one of those is a quick answer to
> > > the question "should I look into this?".
> >
> > Well, there is a link in the project page to the "Overview" section
> > of the wiki, but perhaps is a bit hidden.  I've added some blurb as
> > you suggested in the main page an another link to the "Overview"
> > wiki page. Hope that, by reading the new blurb, you can see why it
> > accelerates expression evaluation with regard to NumPy.  If not,
> > tell me and will try to come with something more comprehensible.
>
> I did see the overview.  The addition you made is great but it's so
> far down that many won't get to it.  Even in its section, the meat of
> it is below three paragraphs that most users won't care about and
> many won't understand.  I've posted some notes on writing intros in
> Developer_Zone.
>
> In the following, I've reordered the page to address the questions of
> potential users first, edited it a bit, and fixed the example to
> conform to our doc standards (and 128->256; hope that was right). 
> See what you think...
[clip]

That's great!  I've heavily changed the docs on the project site.  I've 
followed your advices in most of places, but not always (a `Building` 
section has to be always high on a manual, IMHO).

Thanks a lot for your contribution!

-- 
Francesc Alted


From jonathan.taylor at utoronto.ca  Mon Jan 19 15:43:55 2009
From: jonathan.taylor at utoronto.ca (Jonathan Taylor)
Date: Mon, 19 Jan 2009 15:43:55 -0500
Subject: [Numpy-discussion] Testing for close to zero?
Message-ID: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>

Hi,

When solving a quadratic equation I get that alpha =
-3.78336776728e-31 which I believe to be far below machine precision:

finfo(float).eps
2.2204460492503131e-16

But an if statement like:

if alpha == 0:
   ...

does not catch this.  Is there a better way to check for things that
are essentially zero or should I really be using

if np.abs(alpha) < finfo(float).eps:
   ...

?

Thanks for any help.
Jonathan.


From robert.kern at gmail.com  Mon Jan 19 15:55:39 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 19 Jan 2009 14:55:39 -0600
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
Message-ID: <3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>

On Mon, Jan 19, 2009 at 14:43, Jonathan Taylor
<jonathan.taylor at utoronto.ca> wrote:
> Hi,
>
> When solving a quadratic equation I get that alpha =
> -3.78336776728e-31 which I believe to be far below machine precision:
>
> finfo(float).eps
> 2.2204460492503131e-16
>
> But an if statement like:
>
> if alpha == 0:
>   ...
>
> does not catch this.  Is there a better way to check for things that
> are essentially zero or should I really be using
>
> if np.abs(alpha) < finfo(float).eps:
>   ...

Almost. You should scale eps by some estimate of the size of the
problem. Exactly how you should do this depends on the problem,
though. Errors accumulate in different ways depending on the
operations you perform on the numbers. Multiplying eps by
max(abs(array_of_inputs)) is probably a reasonable starting point.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jh at physics.ucf.edu  Mon Jan 19 16:31:57 2009
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Mon, 19 Jan 2009 16:31:57 -0500
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901192104.22227.faltet@pytables.org> (message from Francesc
	Alted on Mon, 19 Jan 2009 21:04:21 +0100)
References: <wl8k58vgxiw.fsf@glup.physics.ucf.edu>
	<200901161704.51777.faltet@pytables.org>
	<wl8tz7wfgyh.fsf@glup.physics.ucf.edu>
	<200901192104.22227.faltet@pytables.org>
Message-ID: <wl8iqobf0du.fsf@glup.physics.ucf.edu>

Thanks!  I think this will help the package attract a lot of users.

A couple of housekeeping things:

on http://code.google.com/p/numexpr:

  What it is? -> What is it? or What it is (no question mark)

on http://code.google.com/p/numexpr/wiki/Overview:

  The last example got incorporated as straight text somehow.

In firefox, the first code example runs into the pastel boxes on the
right for modest-width browsers.  This is a common problem with
firefox, but I think it comes from improper HTML code that IE somehow
deals with, rather than non-standard behavior in firefox.

One thing I'd add is a benchmark example against numpy.  Make it
simple, so that people can copy and modify the benchmark code to test
their own performance improvements.

I added an entry for it on the Topical Software list.  Please check it
out and modify as you see fit

--jh--


From h5py at alfven.org  Mon Jan 19 19:59:03 2009
From: h5py at alfven.org (Andrew Collette)
Date: Mon, 19 Jan 2009 16:59:03 -0800
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901161300.26130.faltet@pytables.org>
References: <200901161300.26130.faltet@pytables.org>
Message-ID: <f125f5e0901191659k60a264a4m5af68f23b2793c2@mail.gmail.com>

Hi Francesc,

Looks like a cool project!  However, I'm not able to achieve the
advertised speed-ups.  I wrote a simple script to try three approaches
to this kind of problem:

1) Native Python code (i.e. will try to do everything at once using temp arrays)
2) Straightforward numexpr evaluation
3) Simple "chunked" evaluation using array.flat views.  (This solves
the memory problem and allows the use of arbitrary Python
expressions).

I've attached the script; here's the output for the expression
"63 + (a*b) + (c**2) + sin(b)"
along with a few combinations of shapes/dtypes.  As expected, using
anything other than "f8" (double) results in a performance penalty.
Surprisingly, it seems that using chunks via array.flat results in
similar performance for f8, and even better performance for other
dtypes.

(100, 100, 100) f4 (average of 10 runs)
Simple:  0.155238199234
Numexpr:  0.278440499306
Chunked:  0.166213512421

(100, 100, 100) f8 (average of 10 runs)
Simple:  0.241649699211
Numexpr:  0.192837905884
Chunked:  0.183888602257

(100, 100, 100, 10) f4 (average of 10 runs)
Simple:  1.56741549969
Numexpr:  3.40679829121
Chunked:  1.83729870319

(100, 100, 100) i4 (average of 10 runs)
Simple:  0.206279683113
Numexpr:  0.210431909561
Chunked:  0.182894086838

FYI, the current tar file (1.1-1) has a glitch related to the VERSION
file; I added to the bug report at google code.

Andrew Collette

On Fri, Jan 16, 2009 at 4:00 AM, Francesc Alted <faltet at pytables.org> wrote:
> ========================
>  Announcing Numexpr 1.1
> ========================
>
> Numexpr is a fast numerical expression evaluator for NumPy.  With it,
> expressions that operate on arrays (like "3*a+4*b") are accelerated
> and use less memory than doing the same calculation in Python.
>
> The expected speed-ups for Numexpr respect to NumPy are between 0.95x
> and 15x, being 3x or 4x typical values.  The strided and unaligned
> case has been optimized too, so if the expresion contains such arrays,
> the speed-up can increase significantly.  Of course, you will need to
> operate with large arrays (typically larger than the cache size of your
> CPU) to see these improvements in performance.
>
> This release is mainly intended to put in sync some of the
> improvements that had the Numexpr version integrated in PyTables.
> So, this standalone version of Numexpr will benefit of the well tested
> PyTables' version that has been in production for more than a year now.
>
> In case you want to know more in detail what has changed in this
> version, have a look at ``RELEASE_NOTES.txt`` in the tarball.
>
>
> Where I can find Numexpr?
> =========================
>
> The project is hosted at Google code in:
>
> http://code.google.com/p/numexpr/
>
>
> Share your experience
> =====================
>
> Let us know of any bugs, suggestions, gripes, kudos, etc. you may
> have.
>
>
> Enjoy!
>
> --
> Francesc Alted
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: exprtest.py
Type: text/x-python
Size: 2077 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090119/ef0d7e50/attachment.py>

From jonathan.taylor at utoronto.ca  Mon Jan 19 21:23:43 2009
From: jonathan.taylor at utoronto.ca (Jonathan Taylor)
Date: Mon, 19 Jan 2009 21:23:43 -0500
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
Message-ID: <463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>

Interesting.  That makes sense and I suppose that also explains why
there is no function to do this sort of thing for you.

Jon.

On Mon, Jan 19, 2009 at 3:55 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Jan 19, 2009 at 14:43, Jonathan Taylor
> <jonathan.taylor at utoronto.ca> wrote:
>> Hi,
>>
>> When solving a quadratic equation I get that alpha =
>> -3.78336776728e-31 which I believe to be far below machine precision:
>>
>> finfo(float).eps
>> 2.2204460492503131e-16
>>
>> But an if statement like:
>>
>> if alpha == 0:
>>   ...
>>
>> does not catch this.  Is there a better way to check for things that
>> are essentially zero or should I really be using
>>
>> if np.abs(alpha) < finfo(float).eps:
>>   ...
>
> Almost. You should scale eps by some estimate of the size of the
> problem. Exactly how you should do this depends on the problem,
> though. Errors accumulate in different ways depending on the
> operations you perform on the numbers. Multiplying eps by
> max(abs(array_of_inputs)) is probably a reasonable starting point.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Mon Jan 19 23:09:00 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 19 Jan 2009 21:09:00 -0700
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
Message-ID: <e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>

On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor <
jonathan.taylor at utoronto.ca> wrote:

> Interesting.  That makes sense and I suppose that also explains why
> there is no function to do this sort of thing for you.
>

A combination of relative and absolute errors is another common solution,
i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In cases like
this relerr is typically eps and abserr tends to be something like 1e-12,
which keeps you from descending towards zero any further than you need to.

Chuck


>
> Jon.
>
> On Mon, Jan 19, 2009 at 3:55 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> > On Mon, Jan 19, 2009 at 14:43, Jonathan Taylor
> > <jonathan.taylor at utoronto.ca> wrote:
> >> Hi,
> >>
> >> When solving a quadratic equation I get that alpha =
> >> -3.78336776728e-31 which I believe to be far below machine precision:
> >>
> >> finfo(float).eps
> >> 2.2204460492503131e-16
> >>
> >> But an if statement like:
> >>
> >> if alpha == 0:
> >>   ...
> >>
> >> does not catch this.  Is there a better way to check for things that
> >> are essentially zero or should I really be using
> >>
> >> if np.abs(alpha) < finfo(float).eps:
> >>   ...
> >
> > Almost. You should scale eps by some estimate of the size of the
> > problem. Exactly how you should do this depends on the problem,
> > though. Errors accumulate in different ways depending on the
> > operations you perform on the numbers. Multiplying eps by
> > max(abs(array_of_inputs)) is probably a reasonable starting point.
> >
>

> > --
> > Robert Kern
> >
> > "I have come to believe that the whole world is an enigma, a harmless
> > enigma that is made terrible by our own mad attempt to interpret it as
> > though it had an underlying truth."
> >  -- Umberto Eco
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090119/b6f85616/attachment.html>

From robert.kern at gmail.com  Mon Jan 19 23:17:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 19 Jan 2009 22:17:02 -0600
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
	<e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
Message-ID: <3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>

On Mon, Jan 19, 2009 at 22:09, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor
> <jonathan.taylor at utoronto.ca> wrote:
>>
>> Interesting.  That makes sense and I suppose that also explains why
>> there is no function to do this sort of thing for you.
>
> A combination of relative and absolute errors is another common solution,
> i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In cases like
> this relerr is typically eps and abserr tends to be something like 1e-12,
> which keeps you from descending towards zero any further than you need to.

I don't think the absolute error term is appropriate in this case. If
all of my inputs are of the size 1e-12, I would expect a result of
1e-14 to be significantly far from 0.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Tue Jan 20 00:36:02 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 19 Jan 2009 22:36:02 -0700
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
	<e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
	<3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>
Message-ID: <e06186140901192136v1b90a40ch1acbfcc00fbfa4de@mail.gmail.com>

On Mon, Jan 19, 2009 at 9:17 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Jan 19, 2009 at 22:09, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor
> > <jonathan.taylor at utoronto.ca> wrote:
> >>
> >> Interesting.  That makes sense and I suppose that also explains why
> >> there is no function to do this sort of thing for you.
> >
> > A combination of relative and absolute errors is another common solution,
> > i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In cases
> like
> > this relerr is typically eps and abserr tends to be something like 1e-12,
> > which keeps you from descending towards zero any further than you need
> to.
>
> I don't think the absolute error term is appropriate in this case. If
> all of my inputs are of the size 1e-12, I would expect a result of
> 1e-14 to be significantly far from 0.
>

Sure, that's why you *chose* constants appropriate to the problem. As to
this case, I don't know what the quadratic is or what methods are being used
to solve it, or even if the methods are appropriate. So the comment was
general and I think many numeric methods for solving equations use some
variant of the combination. For instance, the 1D zero finders in Scipy use
it.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090119/3c58c09b/attachment.html>

From robert.kern at gmail.com  Tue Jan 20 00:48:34 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 19 Jan 2009 23:48:34 -0600
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <e06186140901192136v1b90a40ch1acbfcc00fbfa4de@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
	<e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
	<3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>
	<e06186140901192136v1b90a40ch1acbfcc00fbfa4de@mail.gmail.com>
Message-ID: <3d375d730901192148o7da2cc55s788b9e92987167ae@mail.gmail.com>

On Mon, Jan 19, 2009 at 23:36, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Mon, Jan 19, 2009 at 9:17 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Mon, Jan 19, 2009 at 22:09, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor
>> > <jonathan.taylor at utoronto.ca> wrote:
>> >>
>> >> Interesting.  That makes sense and I suppose that also explains why
>> >> there is no function to do this sort of thing for you.
>> >
>> > A combination of relative and absolute errors is another common
>> > solution,
>> > i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In cases
>> > like
>> > this relerr is typically eps and abserr tends to be something like
>> > 1e-12,
>> > which keeps you from descending towards zero any further than you need
>> > to.
>>
>> I don't think the absolute error term is appropriate in this case. If
>> all of my inputs are of the size 1e-12, I would expect a result of
>> 1e-14 to be significantly far from 0.
>
> Sure, that's why you *chose* constants appropriate to the problem.

But that's what eps*max(abs(array_of_inputs)) is supposed to do.

In the formulation that you are using (e.g. that of
assert_arrays_almost_equal()), the absolute error comes into play when
you are comparing two numbers in ignorance of the processes that
created them. The relative error in that formula is being adjusted by
the size of the two numbers (*not* the inputs to the algorithm). The
two numbers may be close to 0, but the relevant inputs to the
algorithm may be ~1, let's say. In that case, you need the absolute
error term to provide the scale information that is otherwise not
present in the comparison.

But if you know what the inputs to the calculation were, you can
estimate the scale factor for the relative tolerance directly
(rigorously, if you've done the numerical analysis) and the absolute
tolerance is supernumerary.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Tue Jan 20 01:21:19 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 19 Jan 2009 23:21:19 -0700
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <3d375d730901192148o7da2cc55s788b9e92987167ae@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
	<e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
	<3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>
	<e06186140901192136v1b90a40ch1acbfcc00fbfa4de@mail.gmail.com>
	<3d375d730901192148o7da2cc55s788b9e92987167ae@mail.gmail.com>
Message-ID: <e06186140901192221l53f5c6c6pdce54f28663440f@mail.gmail.com>

On Mon, Jan 19, 2009 at 10:48 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Jan 19, 2009 at 23:36, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> > On Mon, Jan 19, 2009 at 9:17 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Mon, Jan 19, 2009 at 22:09, Charles R Harris
> >> <charlesr.harris at gmail.com> wrote:
> >> >
> >> >
> >> > On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor
> >> > <jonathan.taylor at utoronto.ca> wrote:
> >> >>
> >> >> Interesting.  That makes sense and I suppose that also explains why
> >> >> there is no function to do this sort of thing for you.
> >> >
> >> > A combination of relative and absolute errors is another common
> >> > solution,
> >> > i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In cases
> >> > like
> >> > this relerr is typically eps and abserr tends to be something like
> >> > 1e-12,
> >> > which keeps you from descending towards zero any further than you need
> >> > to.
> >>
> >> I don't think the absolute error term is appropriate in this case. If
> >> all of my inputs are of the size 1e-12, I would expect a result of
> >> 1e-14 to be significantly far from 0.
> >
> > Sure, that's why you *chose* constants appropriate to the problem.
>
> But that's what eps*max(abs(array_of_inputs)) is supposed to do.
>
> In the formulation that you are using (e.g. that of
> assert_arrays_almost_equal()), the absolute error comes into play when
> you are comparing two numbers in ignorance of the processes that
> created them. The relative error in that formula is being adjusted by
> the size of the two numbers (*not* the inputs to the algorithm). The
> two numbers may be close to 0, but the relevant inputs to the
> algorithm may be ~1, let's say. In that case, you need the absolute
> error term to provide the scale information that is otherwise not
> present in the comparison.
>
> But if you know what the inputs to the calculation were, you can
> estimate the scale factor for the relative tolerance directly
> (rigorously, if you've done the numerical analysis) and the absolute
> tolerance is supernumerary.


So you do bisection on an oddball curve,  512 iterations later you hit
zero... Or you do numeric integration where there is lots of cancellation.
These problems aren't new and the mixed method for tolerance is quite
standard and has been for many years. I don't see why you want to argue
about it, if you don't like the combined method, set the absolute error to
zero, problem solved.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090119/6193b277/attachment.html>

From robert.kern at gmail.com  Tue Jan 20 01:26:22 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Jan 2009 00:26:22 -0600
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <e06186140901192221l53f5c6c6pdce54f28663440f@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
	<e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
	<3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>
	<e06186140901192136v1b90a40ch1acbfcc00fbfa4de@mail.gmail.com>
	<3d375d730901192148o7da2cc55s788b9e92987167ae@mail.gmail.com>
	<e06186140901192221l53f5c6c6pdce54f28663440f@mail.gmail.com>
Message-ID: <3d375d730901192226t32f35a1dta4409dca68f3656a@mail.gmail.com>

On Tue, Jan 20, 2009 at 00:21, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Mon, Jan 19, 2009 at 10:48 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Mon, Jan 19, 2009 at 23:36, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> > On Mon, Jan 19, 2009 at 9:17 PM, Robert Kern <robert.kern at gmail.com>
>> > wrote:
>> >>
>> >> On Mon, Jan 19, 2009 at 22:09, Charles R Harris
>> >> <charlesr.harris at gmail.com> wrote:
>> >> >
>> >> >
>> >> > On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor
>> >> > <jonathan.taylor at utoronto.ca> wrote:
>> >> >>
>> >> >> Interesting.  That makes sense and I suppose that also explains why
>> >> >> there is no function to do this sort of thing for you.
>> >> >
>> >> > A combination of relative and absolute errors is another common
>> >> > solution,
>> >> > i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In
>> >> > cases
>> >> > like
>> >> > this relerr is typically eps and abserr tends to be something like
>> >> > 1e-12,
>> >> > which keeps you from descending towards zero any further than you
>> >> > need
>> >> > to.
>> >>
>> >> I don't think the absolute error term is appropriate in this case. If
>> >> all of my inputs are of the size 1e-12, I would expect a result of
>> >> 1e-14 to be significantly far from 0.
>> >
>> > Sure, that's why you *chose* constants appropriate to the problem.
>>
>> But that's what eps*max(abs(array_of_inputs)) is supposed to do.
>>
>> In the formulation that you are using (e.g. that of
>> assert_arrays_almost_equal()), the absolute error comes into play when
>> you are comparing two numbers in ignorance of the processes that
>> created them. The relative error in that formula is being adjusted by
>> the size of the two numbers (*not* the inputs to the algorithm). The
>> two numbers may be close to 0, but the relevant inputs to the
>> algorithm may be ~1, let's say. In that case, you need the absolute
>> error term to provide the scale information that is otherwise not
>> present in the comparison.
>>
>> But if you know what the inputs to the calculation were, you can
>> estimate the scale factor for the relative tolerance directly
>> (rigorously, if you've done the numerical analysis) and the absolute
>> tolerance is supernumerary.
>
>
> So you do bisection on an oddball curve,  512 iterations later you hit
> zero... Or you do numeric integration where there is lots of cancellation.
> These problems aren't new and the mixed method for tolerance is quite
> standard and has been for many years. I don't see why you want to argue
> about it, if you don't like the combined method, set the absolute error to
> zero, problem solved.

I think we're talking about different things. I'm talking about the
way to estimate a good value for the absolute error. My
array_of_inputs was not the values that you are comparing to zero, but
the inputs to the algorithm that created the value you are comparing
to zero.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Tue Jan 20 01:47:33 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 19 Jan 2009 23:47:33 -0700
Subject: [Numpy-discussion] Testing for close to zero?
In-Reply-To: <3d375d730901192226t32f35a1dta4409dca68f3656a@mail.gmail.com>
References: <463e11f90901191243u2fc1c50u3110420c2af859d5@mail.gmail.com>
	<3d375d730901191255o13677721v574a69e8670ffad1@mail.gmail.com>
	<463e11f90901191823r4f778efdwd8373025cbc8182b@mail.gmail.com>
	<e06186140901192009s45450e93k313540b6b75d19f8@mail.gmail.com>
	<3d375d730901192017n2a378a5aybd8e239e5e7f97@mail.gmail.com>
	<e06186140901192136v1b90a40ch1acbfcc00fbfa4de@mail.gmail.com>
	<3d375d730901192148o7da2cc55s788b9e92987167ae@mail.gmail.com>
	<e06186140901192221l53f5c6c6pdce54f28663440f@mail.gmail.com>
	<3d375d730901192226t32f35a1dta4409dca68f3656a@mail.gmail.com>
Message-ID: <e06186140901192247l47f99dfcwbdfc3646679db267@mail.gmail.com>

On Mon, Jan 19, 2009 at 11:26 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Jan 20, 2009 at 00:21, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> > On Mon, Jan 19, 2009 at 10:48 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Mon, Jan 19, 2009 at 23:36, Charles R Harris
> >> <charlesr.harris at gmail.com> wrote:
> >> >
> >> > On Mon, Jan 19, 2009 at 9:17 PM, Robert Kern <robert.kern at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Mon, Jan 19, 2009 at 22:09, Charles R Harris
> >> >> <charlesr.harris at gmail.com> wrote:
> >> >> >
> >> >> >
> >> >> > On Mon, Jan 19, 2009 at 7:23 PM, Jonathan Taylor
> >> >> > <jonathan.taylor at utoronto.ca> wrote:
> >> >> >>
> >> >> >> Interesting.  That makes sense and I suppose that also explains
> why
> >> >> >> there is no function to do this sort of thing for you.
> >> >> >
> >> >> > A combination of relative and absolute errors is another common
> >> >> > solution,
> >> >> > i.e., test against relerr*max(abs(array_of_inputs)) + abserr. In
> >> >> > cases
> >> >> > like
> >> >> > this relerr is typically eps and abserr tends to be something like
> >> >> > 1e-12,
> >> >> > which keeps you from descending towards zero any further than you
> >> >> > need
> >> >> > to.
> >> >>
> >> >> I don't think the absolute error term is appropriate in this case. If
> >> >> all of my inputs are of the size 1e-12, I would expect a result of
> >> >> 1e-14 to be significantly far from 0.
> >> >
> >> > Sure, that's why you *chose* constants appropriate to the problem.
> >>
> >> But that's what eps*max(abs(array_of_inputs)) is supposed to do.
> >>
> >> In the formulation that you are using (e.g. that of
> >> assert_arrays_almost_equal()), the absolute error comes into play when
> >> you are comparing two numbers in ignorance of the processes that
> >> created them. The relative error in that formula is being adjusted by
> >> the size of the two numbers (*not* the inputs to the algorithm). The
> >> two numbers may be close to 0, but the relevant inputs to the
> >> algorithm may be ~1, let's say. In that case, you need the absolute
> >> error term to provide the scale information that is otherwise not
> >> present in the comparison.
> >>
> >> But if you know what the inputs to the calculation were, you can
> >> estimate the scale factor for the relative tolerance directly
> >> (rigorously, if you've done the numerical analysis) and the absolute
> >> tolerance is supernumerary.
> >
> >
> > So you do bisection on an oddball curve,  512 iterations later you hit
> > zero... Or you do numeric integration where there is lots of
> cancellation.
> > These problems aren't new and the mixed method for tolerance is quite
> > standard and has been for many years. I don't see why you want to argue
> > about it, if you don't like the combined method, set the absolute error
> to
> > zero, problem solved.
>
> I think we're talking about different things. I'm talking about the
> way to estimate a good value for the absolute error. My
> array_of_inputs was not the values that you are comparing to zero, but
> the inputs to the algorithm that created the value you are comparing
> to zero.
>

Ah. But that won't generally work for polynomials, they are too ill
conditioned with respect to the coefficients. Even quadratics solved using
the standard formula with the +/- can be ill conditioned. And that isn't to
mention that the zeros are scale invariant, i.e., you can multiply the whole
equation by some ginormous number and the zeros will remain the same. It's
fun for a rainy day to check the scale invariance of the zero estimates of
various solution algorithms.

On the other hand, that method of estimating the error might work for
integrals if the result scales with the input parameters.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090119/dfaf1c3e/attachment.html>

From nwagner at iam.uni-stuttgart.de  Tue Jan 20 04:17:03 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 20 Jan 2009 10:17:03 +0100
Subject: [Numpy-discussion] Examples for numpy.genfromtxt
Message-ID: <web-116097567@uni-stuttgart.de>

Hi all,

Where can I find some sophisticated examples for the usage 
of numpy.genfromtxt ?
  

Nils


From faltet at pytables.org  Tue Jan 20 09:13:01 2009
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 20 Jan 2009 15:13:01 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <f125f5e0901191659k60a264a4m5af68f23b2793c2@mail.gmail.com>
References: <200901161300.26130.faltet@pytables.org>
	<f125f5e0901191659k60a264a4m5af68f23b2793c2@mail.gmail.com>
Message-ID: <200901201513.01432.faltet@pytables.org>

A Tuesday 20 January 2009, Andrew Collette escrigu?:
> Hi Francesc,
>
> Looks like a cool project!  However, I'm not able to achieve the
> advertised speed-ups.  I wrote a simple script to try three
> approaches to this kind of problem:
>
> 1) Native Python code (i.e. will try to do everything at once using
> temp arrays) 2) Straightforward numexpr evaluation
> 3) Simple "chunked" evaluation using array.flat views.  (This solves
> the memory problem and allows the use of arbitrary Python
> expressions).
>
> I've attached the script; here's the output for the expression
> "63 + (a*b) + (c**2) + sin(b)"
> along with a few combinations of shapes/dtypes.  As expected, using
> anything other than "f8" (double) results in a performance penalty.
> Surprisingly, it seems that using chunks via array.flat results in
> similar performance for f8, and even better performance for other
> dtypes.
[clip]

Well, there were two issues there.  The first one is that when 
transcendental functions are used (like sin() above), the bottleneck is 
on the CPU instead of memory bandwidth, so numexpr speedups are not so 
high as usual.  The other issue was an actual bug in the numexpr code 
that forced a copy of all multidimensional arrays (I normally only use 
undimensional arrays for doing benchmarks).  This has been fixed in 
trunk (r39).

So, with the fix on, the timings are:

(100, 100, 100) f4 (average of 10 runs)
Simple:  0.0426136016846
Numexpr:  0.11350851059
Chunked:  0.0635252952576
(100, 100, 100) f8 (average of 10 runs)
Simple:  0.119254398346
Numexpr:  0.10092959404
Chunked:  0.128384995461

The speed-up is now a mere 20% (for f8), but at least it is not slower.  
With the patches that recently contributed Georg for using Intel's VML, 
the acceleration is a bit better:

(100, 100, 100) f4 (average of 10 runs)
Simple:  0.0417867898941
Numexpr:  0.0944641113281
Chunked:  0.0636183023453
(100, 100, 100) f8 (average of 10 runs)
Simple:  0.120059680939
Numexpr:  0.0832288980484
Chunked:  0.128114104271

i.e. the speed-up is around 45% (for f8).

Moreover, if I get rid of the sin() function and use the expresion:

"63 + (a*b) + (c**2) + b"

I get:

(100, 100, 100) f4 (average of 10 runs)
Simple:  0.0119329929352
Numexpr:  0.0198570966721
Chunked:  0.0338240146637
(100, 100, 100) f8 (average of 10 runs)
Simple:  0.0255623102188
Numexpr:  0.00832500457764
Chunked:  0.0340095996857

which has a 3.1x speedup (for f8).

> FYI, the current tar file (1.1-1) has a glitch related to the VERSION
> file; I added to the bug report at google code.

Thanks. Will focus on that asap.  Mmm, seems like there is stuff enough 
for another release of numexpr.  I'll try to do it soon.

Cheers,

-- 
Francesc Alted


From fperez.net at gmail.com  Tue Jan 20 13:56:06 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 20 Jan 2009 10:56:06 -0800
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <1e2af89e0901191120k6b2880betb7e43c2607c1336d@mail.gmail.com>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<4971AD57.2000605@mur.at>
	<1e2af89e0901191120k6b2880betb7e43c2607c1336d@mail.gmail.com>
Message-ID: <db6b5ecc0901201056l43f1010fx8112cd216bbaec4c@mail.gmail.com>

On Mon, Jan 19, 2009 at 11:20 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
>> Do you also know how the situation is with sourceforge/launchpad/trac...
>> and other popular hosting systems ?
>> Do they also have these restrictions ?
>
> I've not noticed any problems with sourceforge, nor launchpad - I'm
> using them regularly from here.   You'd hope that was the case for
> launchpad - when my browser started this morning, it reminded me of
> the Ubuntu / Canonical mission - http://www.canonical.com/aboutus
>
>    1)  delivering the world's best free software platform
>    2)  ensuring its availability to everyone

Also, remember that launchpad makes it pretty trivial to set up a bzr
branch off an external SVN or other repo.  So it should  be very easy
to create launchpad projects that track existing google code ones you
are interested in.

Not arguing with your original point, just providing more info on a
viable workaround in the meantime, especially since it's quite likely
that projects already on google code will not switch hosts due to this
issue.

Best,

f


From h5py at alfven.org  Tue Jan 20 14:21:12 2009
From: h5py at alfven.org (Andrew Collette)
Date: Tue, 20 Jan 2009 11:21:12 -0800
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901201513.01432.faltet@pytables.org>
References: <200901161300.26130.faltet@pytables.org>
	<f125f5e0901191659k60a264a4m5af68f23b2793c2@mail.gmail.com>
	<200901201513.01432.faltet@pytables.org>
Message-ID: <f125f5e0901201121l7a95ad56h3930a6660c9d0460@mail.gmail.com>

Works much, much better with the current svn version. :) Numexpr now
outperforms everything except the "simple" technique, and then only
for small data sets.

Along the lines you mentioned I noticed that simply changing from a
shape of (100*100*100,) to (100, 100, 100) results in nearly a factor
of 2 worse performance, a factor which seems constant when changing
the size of the data set.  Is this related to the way numexpr handles
broadcasting rules?  It would seem the memory contents should be
identical for these two cases.

Andrew

On Tue, Jan 20, 2009 at 6:13 AM, Francesc Alted <faltet at pytables.org> wrote:
> A Tuesday 20 January 2009, Andrew Collette escrigu?:
>> Hi Francesc,
>>
>> Looks like a cool project!  However, I'm not able to achieve the
>> advertised speed-ups.  I wrote a simple script to try three
>> approaches to this kind of problem:
>>
>> 1) Native Python code (i.e. will try to do everything at once using
>> temp arrays) 2) Straightforward numexpr evaluation
>> 3) Simple "chunked" evaluation using array.flat views.  (This solves
>> the memory problem and allows the use of arbitrary Python
>> expressions).
>>
>> I've attached the script; here's the output for the expression
>> "63 + (a*b) + (c**2) + sin(b)"
>> along with a few combinations of shapes/dtypes.  As expected, using
>> anything other than "f8" (double) results in a performance penalty.
>> Surprisingly, it seems that using chunks via array.flat results in
>> similar performance for f8, and even better performance for other
>> dtypes.
> [clip]
>
> Well, there were two issues there.  The first one is that when
> transcendental functions are used (like sin() above), the bottleneck is
> on the CPU instead of memory bandwidth, so numexpr speedups are not so
> high as usual.  The other issue was an actual bug in the numexpr code
> that forced a copy of all multidimensional arrays (I normally only use
> undimensional arrays for doing benchmarks).  This has been fixed in
> trunk (r39).
>
> So, with the fix on, the timings are:
>
> (100, 100, 100) f4 (average of 10 runs)
> Simple:  0.0426136016846
> Numexpr:  0.11350851059
> Chunked:  0.0635252952576
> (100, 100, 100) f8 (average of 10 runs)
> Simple:  0.119254398346
> Numexpr:  0.10092959404
> Chunked:  0.128384995461
>
> The speed-up is now a mere 20% (for f8), but at least it is not slower.
> With the patches that recently contributed Georg for using Intel's VML,
> the acceleration is a bit better:
>
> (100, 100, 100) f4 (average of 10 runs)
> Simple:  0.0417867898941
> Numexpr:  0.0944641113281
> Chunked:  0.0636183023453
> (100, 100, 100) f8 (average of 10 runs)
> Simple:  0.120059680939
> Numexpr:  0.0832288980484
> Chunked:  0.128114104271
>
> i.e. the speed-up is around 45% (for f8).
>
> Moreover, if I get rid of the sin() function and use the expresion:
>
> "63 + (a*b) + (c**2) + b"
>
> I get:
>
> (100, 100, 100) f4 (average of 10 runs)
> Simple:  0.0119329929352
> Numexpr:  0.0198570966721
> Chunked:  0.0338240146637
> (100, 100, 100) f8 (average of 10 runs)
> Simple:  0.0255623102188
> Numexpr:  0.00832500457764
> Chunked:  0.0340095996857
>
> which has a 3.1x speedup (for f8).
>
>> FYI, the current tar file (1.1-1) has a glitch related to the VERSION
>> file; I added to the bug report at google code.
>
> Thanks. Will focus on that asap.  Mmm, seems like there is stuff enough
> for another release of numexpr.  I'll try to do it soon.
>
> Cheers,
>
> --
> Francesc Alted
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From pgmdevlist at gmail.com  Tue Jan 20 15:01:30 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 20 Jan 2009 15:01:30 -0500
Subject: [Numpy-discussion] Examples for numpy.genfromtxt
In-Reply-To: <web-116097567@uni-stuttgart.de>
References: <web-116097567@uni-stuttgart.de>
Message-ID: <DE8BF0A1-9830-457E-90E9-3E22DFAFF5E1@gmail.com>

Till I write some proper doc, you can check the examples in tests/ 
test_io (TestFromTxt suitcase)


On Jan 20, 2009, at 4:17 AM, Nils Wagner wrote:

> Hi all,
>
> Where can I find some sophisticated examples for the usage
> of numpy.genfromtxt ?
>
>
> Nils
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From timmichelsen at gmx-topmail.de  Mon Jan 19 13:26:22 2009
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Mon, 19 Jan 2009 19:26:22 +0100
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <4971AD57.2000605@mur.at>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<4971AD57.2000605@mur.at>
Message-ID: <gl5l0c$gg9$1@ger.gmane.org>

Hello,
last year there has been a discussion on this on the OSGEO list about 
the same issue.

You may check oggeo.discuss at Gmane or Nabble for it.

Kind regards,
Timmie



From robert.kern at gmail.com  Tue Jan 20 18:32:16 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Jan 2009 17:32:16 -0600
Subject: [Numpy-discussion] Please don't use google code for hosting
In-Reply-To: <gl5l0c$gg9$1@ger.gmane.org>
References: <1e2af89e0901161607x277d68f6u8f7e897aa5e46c77@mail.gmail.com>
	<4971AD57.2000605@mur.at> <gl5l0c$gg9$1@ger.gmane.org>
Message-ID: <3d375d730901201532o1ffc623fl57fe73b16e8f39d3@mail.gmail.com>

On Mon, Jan 19, 2009 at 12:26, Tim Michelsen
<timmichelsen at gmx-topmail.de> wrote:
> Hello,
> last year there has been a discussion on this on the OSGEO list about
> the same issue.
>
> You may check oggeo.discuss at Gmane or Nabble for it.

I must say that Wilfred L. Guerin's opinions on the subject were quite
entertaining and either the result of paranoid delusions or a certain
Mark V. Shaney.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Tue Jan 20 21:09:28 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 20 Jan 2009 21:09:28 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
Message-ID: <gl604p$c9i$1@ger.gmane.org>

I tried a little experiment, implementing some code in numpy (usually I 
build modules in c++ to interface to python).  Since these operations are 
all large vectors, I hoped it would be reasonably efficient.

The code in question is simple.  It is a model of an amplifier, modeled by 
it's AM/AM and AM/PM characteristics.

The function in question is the __call__ operator.  The test program plots a 
spectrum, calling this operator 1024 times each time with a vector of 4096.

Any ideas?  The code is not too big, so I'll try to attach it.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ampl.py
Type: text/x-python
Size: 2961 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090120/ff89353e/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linear_interp.py
Type: text/x-python
Size: 851 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090120/ff89353e/attachment-0001.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot_spectrum.py
Type: text/x-python
Size: 4618 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090120/ff89353e/attachment-0002.py>

From robert.kern at gmail.com  Tue Jan 20 21:16:39 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Jan 2009 20:16:39 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl604p$c9i$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
Message-ID: <3d375d730901201816k1c2d3d24v759f1ff34f1eb0b3@mail.gmail.com>

2009/1/20 Neal Becker <ndbecker2 at gmail.com>:
> I tried a little experiment, implementing some code in numpy (usually I
> build modules in c++ to interface to python).  Since these operations are
> all large vectors, I hoped it would be reasonably efficient.
>
> The code in question is simple.  It is a model of an amplifier, modeled by
> it's AM/AM and AM/PM characteristics.
>
> The function in question is the __call__ operator.  The test program plots a
> spectrum, calling this operator 1024 times each time with a vector of 4096.
>
> Any ideas?  The code is not too big, so I'll try to attach it.

Any chance you can make it self-contained?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Tue Jan 20 21:22:52 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Jan 2009 20:22:52 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl604p$c9i$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
Message-ID: <3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>

2009/1/20 Neal Becker <ndbecker2 at gmail.com>:
> I tried a little experiment, implementing some code in numpy (usually I
> build modules in c++ to interface to python).  Since these operations are
> all large vectors, I hoped it would be reasonably efficient.
>
> The code in question is simple.  It is a model of an amplifier, modeled by
> it's AM/AM and AM/PM characteristics.
>
> The function in question is the __call__ operator.  The test program plots a
> spectrum, calling this operator 1024 times each time with a vector of 4096.

If you want to find out what lines in that function are taking the
most time, you can try my line_profiler module:

http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/

That might give us a better idea in the absence of a self-contained example.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Tue Jan 20 21:44:23 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 20 Jan 2009 21:44:23 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
Message-ID: <gl6269$gad$1@ger.gmane.org>

Robert Kern wrote:

> 2009/1/20 Neal Becker <ndbecker2 at gmail.com>:
>> I tried a little experiment, implementing some code in numpy (usually I
>> build modules in c++ to interface to python).  Since these operations are
>> all large vectors, I hoped it would be reasonably efficient.
>>
>> The code in question is simple.  It is a model of an amplifier, modeled
>> by it's AM/AM and AM/PM characteristics.
>>
>> The function in question is the __call__ operator.  The test program
>> plots a spectrum, calling this operator 1024 times each time with a
>> vector of 4096.
> 
> If you want to find out what lines in that function are taking the
> most time, you can try my line_profiler module:
> 
> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
> 
> That might give us a better idea in the absence of a self-contained
> example.
> 
Sounds interesting, I'll give that a try.  But, not sure how to use it.

If my main script is plot_spectrum.py, and I want to profile the 
ampl.__call__ function (defined in ampl.py), what do I need to do?  I tried 
running kernprof.py plot_spectrum.py having added @profile decorators in 
ampl.py, but that didn't work:
  File "../mod/ampl.py", line 43, in ampl
    @profile
NameError: name 'profile' is not defined




From robert.kern at gmail.com  Tue Jan 20 21:52:41 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Jan 2009 20:52:41 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl6269$gad$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl6269$gad$1@ger.gmane.org>
Message-ID: <3d375d730901201852t255389abt3fbc1b6a1594d4ef@mail.gmail.com>

On Tue, Jan 20, 2009 at 20:44, Neal Becker <ndbecker2 at gmail.com> wrote:
> Robert Kern wrote:
>
>> 2009/1/20 Neal Becker <ndbecker2 at gmail.com>:
>>> I tried a little experiment, implementing some code in numpy (usually I
>>> build modules in c++ to interface to python).  Since these operations are
>>> all large vectors, I hoped it would be reasonably efficient.
>>>
>>> The code in question is simple.  It is a model of an amplifier, modeled
>>> by it's AM/AM and AM/PM characteristics.
>>>
>>> The function in question is the __call__ operator.  The test program
>>> plots a spectrum, calling this operator 1024 times each time with a
>>> vector of 4096.
>>
>> If you want to find out what lines in that function are taking the
>> most time, you can try my line_profiler module:
>>
>> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
>>
>> That might give us a better idea in the absence of a self-contained
>> example.
>>
> Sounds interesting, I'll give that a try.  But, not sure how to use it.
>
> If my main script is plot_spectrum.py, and I want to profile the
> ampl.__call__ function (defined in ampl.py), what do I need to do?  I tried
> running kernprof.py plot_spectrum.py having added @profile decorators in
> ampl.py, but that didn't work:
>  File "../mod/ampl.py", line 43, in ampl
>    @profile
> NameError: name 'profile' is not defined

kernprof.py --line-by-line plot_spectrum.py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Tue Jan 20 21:57:13 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 20 Jan 2009 21:57:13 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
Message-ID: <gl62ua$hpt$1@ger.gmane.org>

Robert Kern wrote:

> 2009/1/20 Neal Becker <ndbecker2 at gmail.com>:
>> I tried a little experiment, implementing some code in numpy (usually I
>> build modules in c++ to interface to python).  Since these operations are
>> all large vectors, I hoped it would be reasonably efficient.
>>
>> The code in question is simple.  It is a model of an amplifier, modeled
>> by it's AM/AM and AM/PM characteristics.
>>
>> The function in question is the __call__ operator.  The test program
>> plots a spectrum, calling this operator 1024 times each time with a
>> vector of 4096.
> 
> If you want to find out what lines in that function are taking the
> most time, you can try my line_profiler module:
> 
> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
> 
> That might give us a better idea in the absence of a self-contained
> example.
> 
I see the problem.  Thanks for the great profiler!  You ought to make this 
more widely known.

It seems the big chunks of time are used in data conversion between numpy 
and my own vectors classes.  Mine are wrappers around boost::ublas.  The 
conversion must be falling back on a very inefficient method since there is no 
special code to handle numpy vectors.

Not sure what is the best solution.  It would be _great_ if I could make 
boost::python objects that export a buffer interface, but I have absolutely 
no idea how to do this (and so far noone else has volunteered any info on 
this).




From robert.kern at gmail.com  Tue Jan 20 22:00:14 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Jan 2009 21:00:14 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl62ua$hpt$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
Message-ID: <3d375d730901201900s8e07d1ds3e19435566d02d7@mail.gmail.com>

On Tue, Jan 20, 2009 at 20:57, Neal Becker <ndbecker2 at gmail.com> wrote:

> I see the problem.  Thanks for the great profiler!  You ought to make this
> more widely known.

I'll be making a release shortly.

> It seems the big chunks of time are used in data conversion between numpy
> and my own vectors classes.  Mine are wrappers around boost::ublas.  The
> conversion must be falling back on a very inefficient method since there is no
> special code to handle numpy vectors.
>
> Not sure what is the best solution.  It would be _great_ if I could make
> boost::python objects that export a buffer interface, but I have absolutely
> no idea how to do this (and so far noone else has volunteered any info on
> this).

Who's not volunteering information, boost::python or us?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From tjhnson at gmail.com  Wed Jan 21 02:25:41 2009
From: tjhnson at gmail.com (T J)
Date: Tue, 20 Jan 2009 23:25:41 -0800
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl62ua$hpt$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
Message-ID: <b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>

On Tue, Jan 20, 2009 at 6:57 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> It seems the big chunks of time are used in data conversion between numpy
> and my own vectors classes.  Mine are wrappers around boost::ublas.  The
> conversion must be falling back on a very inefficient method since there is no
> special code to handle numpy vectors.
>
> Not sure what is the best solution.  It would be _great_ if I could make
> boost::python objects that export a buffer interface, but I have absolutely
> no idea how to do this (and so far noone else has volunteered any info on
> this).
>

I'm not sure if I've understood everything here, but I think that
pyublas provides exactly what you need.

http://tiker.net/doc/pyublas/


From robert.kern at gmail.com  Wed Jan 21 03:23:09 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Jan 2009 02:23:09 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl62ua$hpt$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
Message-ID: <3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>

On Tue, Jan 20, 2009 at 20:57, Neal Becker <ndbecker2 at gmail.com> wrote:

> I see the problem.  Thanks for the great profiler!  You ought to make this
> more widely known.

http://pypi.python.org/pypi/line_profiler

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From faltet at pytables.org  Wed Jan 21 06:41:34 2009
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 21 Jan 2009 12:41:34 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <f125f5e0901201121l7a95ad56h3930a6660c9d0460@mail.gmail.com>
References: <200901161300.26130.faltet@pytables.org>
	<200901201513.01432.faltet@pytables.org>
	<f125f5e0901201121l7a95ad56h3930a6660c9d0460@mail.gmail.com>
Message-ID: <200901211241.35007.faltet@pytables.org>

A Tuesday 20 January 2009, Andrew Collette escrigu?:
> Works much, much better with the current svn version. :) Numexpr now
> outperforms everything except the "simple" technique, and then only
> for small data sets.

Correct.  This is because of the cost of parsing the expression and 
initializing the virtual machine.  However, as soon as the sizes of the 
operands exceeds the cache of your processor you are starting to see 
the improvement in performance.

> Along the lines you mentioned I noticed that simply changing from a
> shape of (100*100*100,) to (100, 100, 100) results in nearly a factor
> of 2 worse performance, a factor which seems constant when changing
> the size of the data set.

Sorry, but I cannot reproduce this.  When using the expression:

"63 + (a*b) + (c**2) + b"

I get on my machine (Core2 at 3 GHz, running openSUSE Linux 11.1):

1000000 f8 (average of 10 runs)
Simple:  0.0278068065643
Numexpr:  0.00839750766754
Chunked:  0.0266514062881

(100, 100, 100) f8 (average of 10 runs)
Simple:  0.0277318000793
Numexpr:  0.00851640701294
Chunked:  0.0346593856812

and these are the expected results (i.e. no change in performance due to 
multidimensional arrays).  Even for larger arrays, I don't see nothing 
unexpected:

10000000 f8 (average of 10 runs)
Simple:  0.334054994583
Numexpr:  0.110022115707
Chunked:  0.29678030014

(100, 100, 100, 10) f8 (average of 10 runs)
Simple:  0.339299607277
Numexpr:  0.111632704735
Chunked:  0.375299096107

Can you tell us which platforms are you using?

> Is this related to the way numexpr handles 
> broadcasting rules?  It would seem the memory contents should be
> identical for these two cases.
>
> Andrew
>
> On Tue, Jan 20, 2009 at 6:13 AM, Francesc Alted <faltet at pytables.org> 
wrote:
> > A Tuesday 20 January 2009, Andrew Collette escrigu?:
> >> Hi Francesc,
> >>
> >> Looks like a cool project!  However, I'm not able to achieve the
> >> advertised speed-ups.  I wrote a simple script to try three
> >> approaches to this kind of problem:
> >>
> >> 1) Native Python code (i.e. will try to do everything at once
> >> using temp arrays) 2) Straightforward numexpr evaluation
> >> 3) Simple "chunked" evaluation using array.flat views.  (This
> >> solves the memory problem and allows the use of arbitrary Python
> >> expressions).
> >>
> >> I've attached the script; here's the output for the expression
> >> "63 + (a*b) + (c**2) + sin(b)"
> >> along with a few combinations of shapes/dtypes.  As expected,
> >> using anything other than "f8" (double) results in a performance
> >> penalty. Surprisingly, it seems that using chunks via array.flat
> >> results in similar performance for f8, and even better performance
> >> for other dtypes.
> >
> > [clip]
> >
> > Well, there were two issues there.  The first one is that when
> > transcendental functions are used (like sin() above), the
> > bottleneck is on the CPU instead of memory bandwidth, so numexpr
> > speedups are not so high as usual.  The other issue was an actual
> > bug in the numexpr code that forced a copy of all multidimensional
> > arrays (I normally only use undimensional arrays for doing
> > benchmarks).  This has been fixed in trunk (r39).
> >
> > So, with the fix on, the timings are:
> >
> > (100, 100, 100) f4 (average of 10 runs)
> > Simple:  0.0426136016846
> > Numexpr:  0.11350851059
> > Chunked:  0.0635252952576
> > (100, 100, 100) f8 (average of 10 runs)
> > Simple:  0.119254398346
> > Numexpr:  0.10092959404
> > Chunked:  0.128384995461
> >
> > The speed-up is now a mere 20% (for f8), but at least it is not
> > slower. With the patches that recently contributed Georg for using
> > Intel's VML, the acceleration is a bit better:
> >
> > (100, 100, 100) f4 (average of 10 runs)
> > Simple:  0.0417867898941
> > Numexpr:  0.0944641113281
> > Chunked:  0.0636183023453
> > (100, 100, 100) f8 (average of 10 runs)
> > Simple:  0.120059680939
> > Numexpr:  0.0832288980484
> > Chunked:  0.128114104271
> >
> > i.e. the speed-up is around 45% (for f8).
> >
> > Moreover, if I get rid of the sin() function and use the expresion:
> >
> > "63 + (a*b) + (c**2) + b"
> >
> > I get:
> >
> > (100, 100, 100) f4 (average of 10 runs)
> > Simple:  0.0119329929352
> > Numexpr:  0.0198570966721
> > Chunked:  0.0338240146637
> > (100, 100, 100) f8 (average of 10 runs)
> > Simple:  0.0255623102188
> > Numexpr:  0.00832500457764
> > Chunked:  0.0340095996857
> >
> > which has a 3.1x speedup (for f8).
> >
> >> FYI, the current tar file (1.1-1) has a glitch related to the
> >> VERSION file; I added to the bug report at google code.
> >
> > Thanks. Will focus on that asap.  Mmm, seems like there is stuff
> > enough for another release of numexpr.  I'll try to do it soon.
> >
> > Cheers,
> >
> > --
> > Francesc Alted
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



-- 
Francesc Alted


From malkarouri at yahoo.co.uk  Wed Jan 21 06:55:36 2009
From: malkarouri at yahoo.co.uk (Muhammad Alkarouri)
Date: Wed, 21 Jan 2009 11:55:36 +0000 (GMT)
Subject: [Numpy-discussion] numpy and bitwise arrays?
Message-ID: <923458.60721.qm@web27904.mail.ukl.yahoo.com>

Hi everyone,

I need to use a bitwise array, and I wanted to check what is the common practice using numpy.

I expect to read binary strings (like '0011000001') of equal length from file and to save and manipulate them using numpy. I know that numpy have good implementations of bitwise operations, but I was wondering how to encode the data in the first place. As a Python long? As an int ndarray? Which int type? How to save and load such data in such a way that when I load it the information that it is a bit array is stored?


Any help would be appreciated.


Regards,

Muhammad Alkarouri


      


From ndbecker2 at gmail.com  Wed Jan 21 07:27:04 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 07:27:04 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>
Message-ID: <gl74ao$62o$1@ger.gmane.org>

T J wrote:

> On Tue, Jan 20, 2009 at 6:57 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> It seems the big chunks of time are used in data conversion between numpy
>> and my own vectors classes.  Mine are wrappers around boost::ublas.  The
>> conversion must be falling back on a very inefficient method since there
>> is no special code to handle numpy vectors.
>>
>> Not sure what is the best solution.  It would be _great_ if I could make
>> boost::python objects that export a buffer interface, but I have
>> absolutely no idea how to do this (and so far noone else has volunteered
>> any info on this).
>>
> 
> I'm not sure if I've understood everything here, but I think that
> pyublas provides exactly what you need.
> 
> http://tiker.net/doc/pyublas/

It might if I had used this for all of my c++ code, but I have a big library 
of c++ wrapped code that doesn't use pyublas.  Pyublas takes numpy objects 
from python and allows the use of c++ ublas on it (without conversion).

Most of my code doesn't use numpy, it uses plain ublas to represent vectors, 
and ublas handles storage.  I can only interface to/from numpy with 
conversion.

I'm interested in pyublas, but devel seems very quiet for a while.




From nwagner at iam.uni-stuttgart.de  Wed Jan 21 07:29:18 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Wed, 21 Jan 2009 13:29:18 +0100
Subject: [Numpy-discussion] savetxt and blanks
Message-ID: <web-116153110@uni-stuttgart.de>

Hi all,

Is it possible to add a certain number of blanks behind 
the
second column in connection with savetxt ?

from numpy.random import rand
from numpy import savetxt
A = rand(100,2)
savetxt('noblanks.dat',A,fmt='%10.2f %10.2f')

Nils
  


From ndbecker2 at gmail.com  Wed Jan 21 07:28:50 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 07:28:50 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901201900s8e07d1ds3e19435566d02d7@mail.gmail.com>
Message-ID: <gl74e2$62o$2@ger.gmane.org>

Robert Kern wrote:

> On Tue, Jan 20, 2009 at 20:57, Neal Becker <ndbecker2 at gmail.com> wrote:
> 
>> I see the problem.  Thanks for the great profiler!  You ought to make
>> this more widely known.
> 
> I'll be making a release shortly.
> 
>> It seems the big chunks of time are used in data conversion between numpy
>> and my own vectors classes.  Mine are wrappers around boost::ublas.  The
>> conversion must be falling back on a very inefficient method since there
>> is no special code to handle numpy vectors.
>>
>> Not sure what is the best solution.  It would be _great_ if I could make
>> boost::python objects that export a buffer interface, but I have
>> absolutely no idea how to do this (and so far noone else has volunteered
>> any info on this).
> 
> Who's not volunteering information, boost::python or us?
> 
I've asked on python.c++, the home of boost::python and friends.  I've spent 
a little time on it myself, but I think this job requires great knowledge of 
python c api as well as the mysteries of boost::python.




From stefan at sun.ac.za  Wed Jan 21 07:49:32 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 21 Jan 2009 14:49:32 +0200
Subject: [Numpy-discussion] numpy and bitwise arrays?
In-Reply-To: <923458.60721.qm@web27904.mail.ukl.yahoo.com>
References: <923458.60721.qm@web27904.mail.ukl.yahoo.com>
Message-ID: <9457e7c80901210449n57e65d0awcd259f33f4ba1923@mail.gmail.com>

Hi Muhammad

2009/1/21 Muhammad Alkarouri <malkarouri at yahoo.co.uk>:
> I need to use a bitwise array, and I wanted to check what is the common practice using numpy.

You can also take a look at Ilan Schnell's bitarray:

http://pypi.python.org/pypi/bitarray/

Cheers
St?fan


From sturla at molden.no  Wed Jan 21 08:38:34 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Jan 2009 14:38:34 +0100
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl74ao$62o$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>	<gl62ua$hpt$1@ger.gmane.org>	<b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>
	<gl74ao$62o$1@ger.gmane.org>
Message-ID: <4977255A.8010404@molden.no>

On 1/21/2009 1:27 PM, Neal Becker wrote:

> It might if I had used this for all of my c++ code, but I have a big library 
> of c++ wrapped code that doesn't use pyublas.  Pyublas takes numpy objects 
> from python and allows the use of c++ ublas on it (without conversion).

If you can get a pointer (as integer) to your C++ data, and the shape 
and dtype is known, you may use this (rather unsafe) 'fromaddress' hack:


http://www.mail-archive.com/numpy-discussion at scipy.org/msg04974.html


import numpy
def fromaddress(address, dtype, shape, strides=None):
     """
     Create a numpy array from an integer address, a dtype
     or dtype string, a shape tuple, and possibly strides.
     Make sure dtype is a dtype, not just "f" or whatever.
     """
     dtype = numpy.dtype(dtype)
     class Dummy(object): pass
     d = Dummy()
     d.__array_interface__ = dict(
         data = (address, False),
         typestr = dtype.str,
         descr = dtype.descr,
         shape = shape,
         strides = strides,
         version = 3,
     )
     return numpy.asarray(d)




Example:

 >>> a = numpy.zeros(10)
 >>> a
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
 >>> a.__array_interface__
{'descr': [('', '<f8')], 'strides': None, 'shape': (10,), 'version': 3, 
'typestr': '<f8', 'data': (20388752, False)}
 >>> b = fromaddress(20388752, numpy.float64, (10,))
 >>> b
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
 >>> b[0] = 1.0
 >>> a
array([ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])



Sturla Molden




From sturla at molden.no  Wed Jan 21 08:45:36 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Jan 2009 14:45:36 +0100
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <4977255A.8010404@molden.no>
References: <gl604p$c9i$1@ger.gmane.org>	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>	<gl62ua$hpt$1@ger.gmane.org>	<b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>	<gl74ao$62o$1@ger.gmane.org>
	<4977255A.8010404@molden.no>
Message-ID: <49772700.8030504@molden.no>

On 1/21/2009 2:38 PM, Sturla Molden wrote:

> If you can get a pointer (as integer) to your C++ data, and the shape 
> and dtype is known, you may use this (rather unsafe) 'fromaddress' hack:

And opposite, if you need to get the address referenced to by an 
ndarray, you can do this:

def addressof(array):
    return arr.__array_interface__['data'][0]

Then you will have to cast this unsigned integer to a pointer type in 
C++. Note that arr.data returns a buffer.


Sturla Molden



From lists_ravi at lavabit.com  Wed Jan 21 09:37:36 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 21 Jan 2009 09:37:36 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl74ao$62o$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>
	<gl74ao$62o$1@ger.gmane.org>
Message-ID: <200901210937.36222.lists_ravi@lavabit.com>

Hi Neal,

On Wednesday 21 January 2009 07:27:04 Neal Becker wrote:
> It might if I had used this for all of my c++ code, but I have a big
> library of c++ wrapped code that doesn't use pyublas. ?Pyublas takes numpy
> objects from python and allows the use of c++ ublas on it (without
> conversion).
>
> Most of my code doesn't use numpy, it uses plain ublas to represent
> vectors, and ublas handles storage. ?I can only interface to/from numpy
> with conversion.

I pointed out my code to you on c++-sig[1] a while back that solves precisely 
this problem. You found a bug with memory management that I fixed in the 
updated code. Does that still not work for you?

Regards,
Ravi

[1] http://mail.python.org/pipermail/cplusplus-sig/2008-October/013825.html




From malkarouri at yahoo.co.uk  Wed Jan 21 10:21:24 2009
From: malkarouri at yahoo.co.uk (Muhammad Alkarouri)
Date: Wed, 21 Jan 2009 15:21:24 +0000 (GMT)
Subject: [Numpy-discussion] numpy and bitwise arrays?
In-Reply-To: <9457e7c80901210449n57e65d0awcd259f33f4ba1923@mail.gmail.com>
Message-ID: <752361.79246.qm@web27903.mail.ukl.yahoo.com>

--- On Wed, 21/1/09, St?fan van der Walt <stefan at sun.ac.za> wrote:

> From: St?fan van der Walt <stefan at sun.ac.za>
...
> You can also take a look at Ilan Schnell's bitarray:
> 
> http://pypi.python.org/pypi/bitarray/

Looks good to me. Thanks for the suggestion.

Muhammad


      


From ndbecker2 at gmail.com  Wed Jan 21 10:22:36 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 10:22:36 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>
	<gl74ao$62o$1@ger.gmane.org>
	<200901210937.36222.lists_ravi@lavabit.com>
Message-ID: <gl7ejs$do8$1@ger.gmane.org>

Ravi wrote:

> Hi Neal,
> 
> On Wednesday 21 January 2009 07:27:04 Neal Becker wrote:
>> It might if I had used this for all of my c++ code, but I have a big
>> library of c++ wrapped code that doesn't use pyublas. ?Pyublas takes
>> numpy objects from python and allows the use of c++ ublas on it (without
>> conversion).
>>
>> Most of my code doesn't use numpy, it uses plain ublas to represent
>> vectors, and ublas handles storage. ?I can only interface to/from numpy
>> with conversion.
> 
> I pointed out my code to you on c++-sig[1] a while back that solves
> precisely this problem. You found a bug with memory management that I
> fixed in the updated code. Does that still not work for you?
> 
> Regards,
> Ravi
> 
> [1]
> [http://mail.python.org/pipermail/cplusplus-sig/2008-October/013825.html
Thanks for reminding me about this!

Do you have a current version of the code?  I grabbed the files from the 
above message, but I see some additional subsequent messages with more 
patches.




From ndbecker2 at gmail.com  Wed Jan 21 10:38:44 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 10:38:44 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<b02c60870901202325v4dde6b6cs8438c678fc06f9a7@mail.gmail.com>
	<gl74ao$62o$1@ger.gmane.org>
	<200901210937.36222.lists_ravi@lavabit.com>
Message-ID: <gl7fi5$g5p$1@ger.gmane.org>

Ravi wrote:

> Hi Neal,
> 
> On Wednesday 21 January 2009 07:27:04 Neal Becker wrote:
>> It might if I had used this for all of my c++ code, but I have a big
>> library of c++ wrapped code that doesn't use pyublas. ?Pyublas takes
>> numpy objects from python and allows the use of c++ ublas on it (without
>> conversion).
>>
>> Most of my code doesn't use numpy, it uses plain ublas to represent
>> vectors, and ublas handles storage. ?I can only interface to/from numpy
>> with conversion.
> 
> I pointed out my code to you on c++-sig[1] a while back that solves
> precisely this problem. You found a bug with memory management that I
> fixed in the updated code. Does that still not work for you?
> 
> Regards,
> Ravi
> 
> [1]
> [http://mail.python.org/pipermail/cplusplus-sig/2008-October/013825.html

Do you know if this code will work with strided vectors?  If I pass a slice:

u = array (...)
F (u[::2])

What happens?




From oliphant at enthought.com  Wed Jan 21 10:41:45 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 21 Jan 2009 09:41:45 -0600
Subject: [Numpy-discussion] numpy and bitwise arrays?
In-Reply-To: <752361.79246.qm@web27903.mail.ukl.yahoo.com>
References: <752361.79246.qm@web27903.mail.ukl.yahoo.com>
Message-ID: <49774239.4020609@enthought.com>

Muhammad Alkarouri wrote:
> --- On Wed, 21/1/09, St?fan van der Walt <stefan at sun.ac.za> wrote:
>
>   
>> From: St?fan van der Walt <stefan at sun.ac.za>
>>     
> ...
>   
>> You can also take a look at Ilan Schnell's bitarray:
>>
>> http://pypi.python.org/pypi/bitarray/
>>     
>
> Looks good to me. Thanks for the suggestion.
>   

You might also make use of the NumPy functions:

packbits
unpackbits
fromfile

Read the bits in as uint8 data using fromfile.

Then, you can manipulate them either using bit twiddling or with 
indexing operations after unpacking to boolean arrays.


-Travis



From lists_ravi at lavabit.com  Wed Jan 21 11:30:03 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 21 Jan 2009 11:30:03 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl7ejs$do8$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<200901210937.36222.lists_ravi@lavabit.com>
	<gl7ejs$do8$1@ger.gmane.org>
Message-ID: <200901211130.04473.lists_ravi@lavabit.com>

On Wednesday 21 January 2009 10:22:36 Neal Becker wrote:
> > [http://mail.python.org/pipermail/cplusplus-sig/2008-October/013825.html
>
> Thanks for reminding me about this!
>
> Do you have a current version of the code? ?I grabbed the files from the
> above message, but I see some additional subsequent messages with more
> patches.

That is the latest publicly posted code. Since then, there is just one minor 
patch (attached) which enables use of row-major (c-contiguous) arrays.

This does *not* work with strided arrays which would be a fair bit of effort 
to support. Further, you will have to work with the numpy iterator interface, 
which, while well-designed, is a great illustration of the effort required to 
support OO programming in an non-OO language, and is pretty tedious to map to 
the ublas storage iterator interface. If you do implement it, I would very 
much like to take a look at it.

Regards,
Ravi

-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_column_major.patch
Type: text/x-patch
Size: 5446 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/111930bb/attachment.bin>

From dsdale24 at gmail.com  Wed Jan 21 11:34:07 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 21 Jan 2009 11:34:07 -0500
Subject: [Numpy-discussion] strange multiplication behavior with
	numpy.float64 and ndarray subclass
Message-ID: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>

I have a simple test script here that multiplies an ndarray subclass with
another number. Can anyone help me understand why each of these combinations
returns a new instance of MyArray:

mine = MyArray()
print type(np.float32(1)*mine)
print type(mine*np.float32(1))
print type(mine*np.float64(1))
print type(1*mine)
print type(mine*1)

but this one returns a np.float64 instance?

print type(np.float64(1)*mine)


Here is the full script:

import numpy as np

class MyArray(np.ndarray):

    __array_priority__ = 20

    def __new__(cls):
        return np.asarray(1).view(cls).copy()

    def __repr__(self):
        return 'my_array'

    __str__ = __repr__

    def __mul__(self, other):
        return super(MyArray, self).__mul__(other)

    def __rmul__(self, other):
        return super(MyArray, self).__rmul__(other)

mine = MyArray()
print type(np.float32(1)*mine)
print type(mine*np.float32(1))
print type(mine*np.float64(1))
print type(1*mine)
print type(mine*1)

print type(np.float64(1)*mine)


Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/52c6f7fb/attachment.html>

From pgmdevlist at gmail.com  Wed Jan 21 11:43:53 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 21 Jan 2009 11:43:53 -0500
Subject: [Numpy-discussion] strange multiplication behavior with
	numpy.float64 and ndarray subclass
In-Reply-To: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>
References: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>
Message-ID: <CCA1373B-4E2E-421D-94BC-2F5B907C4411@gmail.com>


On Jan 21, 2009, at 11:34 AM, Darren Dale wrote:

> I have a simple test script here that multiplies an ndarray subclass  
> with another number. Can anyone help me understand why each of these  
> combinations returns a new instance of MyArray:
>
> mine = MyArray()
> print type(np.float32(1)*mine)
> print type(mine*np.float32(1))
> print type(mine*np.float64(1))
> print type(1*mine)
> print type(mine*1)
>
> but this one returns a np.float64 instance?

FYI, that's the same behavior as observed in ticket #826. A first  
thread addressed that issue
http://www.mail-archive.com/numpy-discussion at scipy.org/msg13235.html
But so far, no answer has been suggested.
Any help welcome.


From dsdale24 at gmail.com  Wed Jan 21 12:05:23 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 21 Jan 2009 12:05:23 -0500
Subject: [Numpy-discussion] strange multiplication behavior with
	numpy.float64 and ndarray subclass
In-Reply-To: <CCA1373B-4E2E-421D-94BC-2F5B907C4411@gmail.com>
References: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>
	<CCA1373B-4E2E-421D-94BC-2F5B907C4411@gmail.com>
Message-ID: <a08e5f80901210905k3ca43f97sacfb492456ec2359@mail.gmail.com>

On Wed, Jan 21, 2009 at 11:43 AM, Pierre GM <pgmdevlist at gmail.com> wrote:

>
> On Jan 21, 2009, at 11:34 AM, Darren Dale wrote:
>
> > I have a simple test script here that multiplies an ndarray subclass
> > with another number. Can anyone help me understand why each of these
> > combinations returns a new instance of MyArray:
> >
> > mine = MyArray()
> > print type(np.float32(1)*mine)
> > print type(mine*np.float32(1))
> > print type(mine*np.float64(1))
> > print type(1*mine)
> > print type(mine*1)
> >
> > but this one returns a np.float64 instance?
>
> FYI, that's the same behavior as observed in ticket #826. A first
> thread addressed that issue
> http://www.mail-archive.com/numpy-discussion at scipy.org/msg13235.html
> But so far, no answer has been suggested.
> Any help welcome.


I dont understand why __array_priority__ is not being respected here. Ticket
826 lists the component as numpy.ma, it seems the problem is in numpy.core.
I think the severity of the ticket should be increased. But I wasnt able to
view the ticket, I keep getting an "internal server error".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/c5c8534d/attachment.html>

From pgmdevlist at gmail.com  Wed Jan 21 12:26:22 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 21 Jan 2009 12:26:22 -0500
Subject: [Numpy-discussion] strange multiplication behavior with
	numpy.float64 and ndarray subclass
In-Reply-To: <a08e5f80901210905k3ca43f97sacfb492456ec2359@mail.gmail.com>
References: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>
	<CCA1373B-4E2E-421D-94BC-2F5B907C4411@gmail.com>
	<a08e5f80901210905k3ca43f97sacfb492456ec2359@mail.gmail.com>
Message-ID: <F3450F13-AC5D-4000-8812-919C303CE797@gmail.com>

> I dont understand why __array_priority__ is not being respected  
> here. Ticket 826 lists the component as numpy.ma, it seems the  
> problem is in numpy.core. I think the severity of the ticket should  
> be increased. But I wasnt able to view the ticket, I keep getting an  
> "internal server error".

Ticket #826 bumped.




From Chris.Barker at noaa.gov  Wed Jan 21 12:44:29 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 21 Jan 2009 09:44:29 -0800
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl604p$c9i$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
Message-ID: <49775EFD.9000806@noaa.gov>

Neal Becker wrote:
> I tried a little experiment, implementing some code in numpy 

It sounds like you've found your core issue, but a couple comments:

 > from numpy import *

I'm convinced that "import *" is a bad idea. I think the "standard" 
syntax is now "import numpy as np"

 > from math import pi

numpy already has pi -- I find I never need math, if I'm using numpy.



def db_to_volt (db):
     return 10**(0.05*db)

...

class ampl (object):
...
         ampl_interp = linear_interp (vectorize (db_to_volt) (pin), 
db_to_volt (pout))


you shouldn't need to use vectorize here -- db_to_volt already takes 
array input. vectorize could kill performance, in fact.

ampl_interp = linear_interp(db_to_volt(pin), db_to_volt(pout))

should work fine.

also, if you want maximum performance, you can eliminate extraneous 
array creation in functions like that by:

1) using numexpr (see recent posts about it)
2) writing uglier code that explicitly passes in the output arrays:

def db_to_volt (db):
     a = 0.05*db
     np.power(10, a, a)

This will only help for large arrays, and help more for more complex 
functions.


A minor style nit:

I found it remarkably hard to read your code because of the spaces 
before the open parens for function calls:

func (arg1, arg2)

It's not just me: PEP 8 makes it very clear:

"""
Whitespace in Expressions and Statements

   Pet Peeves

     Avoid extraneous whitespace in the following situations:

     - Immediately before the open parenthesis that starts the argument
       list of a function call:

       Yes: spam(1)
       No:  spam (1)
"""

http://www.python.org/dev/peps/pep-0008/

I image you've used that style for years for lots of code, but I 
couldn't help myself!

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From dsdale24 at gmail.com  Wed Jan 21 13:07:59 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 21 Jan 2009 13:07:59 -0500
Subject: [Numpy-discussion] strange multiplication behavior with
	numpy.float64 and ndarray subclass
In-Reply-To: <F3450F13-AC5D-4000-8812-919C303CE797@gmail.com>
References: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>
	<CCA1373B-4E2E-421D-94BC-2F5B907C4411@gmail.com>
	<a08e5f80901210905k3ca43f97sacfb492456ec2359@mail.gmail.com>
	<F3450F13-AC5D-4000-8812-919C303CE797@gmail.com>
Message-ID: <a08e5f80901211007m6be7cc07qff9f8888d8182a87@mail.gmail.com>

On Wed, Jan 21, 2009 at 12:26 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

> > I dont understand why __array_priority__ is not being respected
> > here. Ticket 826 lists the component as numpy.ma, it seems the
> > problem is in numpy.core. I think the severity of the ticket should
> > be increased. But I wasnt able to view the ticket, I keep getting an
> > "internal server error".
>
> Ticket #826 bumped.
>

Just an additional bit of context. I'm working on a subclass that handles
physical quantities, and issue 826 causes a quantity to be converted to a
dimensionless magnitude.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/1c3289d4/attachment.html>

From wesmckinn at gmail.com  Wed Jan 21 13:13:00 2009
From: wesmckinn at gmail.com (Wes McKinney)
Date: Wed, 21 Jan 2009 13:13:00 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
Message-ID: <6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>

Robert-- this is a great little piece of code, I already think it will be a
part of my workflow. However, I seem to be getting negative % time taken on
the more time consuming lines, perhaps getting some overflow?

Thanks a lot,
Wes

On Wed, Jan 21, 2009 at 3:23 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Jan 20, 2009 at 20:57, Neal Becker <ndbecker2 at gmail.com> wrote:
>
> > I see the problem.  Thanks for the great profiler!  You ought to make
> this
> > more widely known.
>
> http://pypi.python.org/pypi/line_profiler
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/a0a40ebe/attachment.html>

From simpson at math.toronto.edu  Wed Jan 21 13:53:15 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Wed, 21 Jan 2009 13:53:15 -0500
Subject: [Numpy-discussion] failure
Message-ID: <CB76AAD1-9AFA-4904-916B-FC8DA13CFB85@math.toronto.edu>

Installing on a Sun machine with Red Hat linux, I got the following  
error:

======================================================================
FAIL: test_umath.TestComplexFunctions.test_against_cmath
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/nose/ 
case.py", line 182, in runTest
     self.test(*self.arg)
   File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/ 
numpy/core/tests/test_umath.py", line 268, in test_against_cmath
     assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b)
AssertionError: arcsinh -2j: (-1.31695789692-1.57079632679j); cmath:  
(1.31695789692-1.57079632679j)

----------------------------------------------------------------------
Ran 1740 tests in 9.839s

FAILED (KNOWNFAIL=1, failures=1)
<nose.result.TextTestResult run=1740 errors=0 failures=1>

How would you recommend I troubleshoot this?  How seriously should I  
take it?

This is with a fresh Python 2.5.4 installation too.

-gideon



From h5py at alfven.org  Wed Jan 21 13:55:39 2009
From: h5py at alfven.org (Andrew Collette)
Date: Wed, 21 Jan 2009 10:55:39 -0800
Subject: [Numpy-discussion] ANN: Numexpr 1.1,
	an efficient array evaluator
In-Reply-To: <200901211241.35007.faltet@pytables.org>
References: <200901161300.26130.faltet@pytables.org>
	<200901201513.01432.faltet@pytables.org>
	<f125f5e0901201121l7a95ad56h3930a6660c9d0460@mail.gmail.com>
	<200901211241.35007.faltet@pytables.org>
Message-ID: <f125f5e0901211055k7da5963dr60cf8cba1cc44955@mail.gmail.com>

Hi,

I get identical results for both shapes now; I manually removed the
"numexpr-1.1.1.dev-py2.5-linux-i686.egg" folder in site-packages and
reinstalled.  I suppose there must have been a stale set of files
somewhere.

Andrew Collette

On Wed, Jan 21, 2009 at 3:41 AM, Francesc Alted <faltet at pytables.org> wrote:
> A Tuesday 20 January 2009, Andrew Collette escrigu?:
>> Works much, much better with the current svn version. :) Numexpr now
>> outperforms everything except the "simple" technique, and then only
>> for small data sets.
>
> Correct.  This is because of the cost of parsing the expression and
> initializing the virtual machine.  However, as soon as the sizes of the
> operands exceeds the cache of your processor you are starting to see
> the improvement in performance.
>
>> Along the lines you mentioned I noticed that simply changing from a
>> shape of (100*100*100,) to (100, 100, 100) results in nearly a factor
>> of 2 worse performance, a factor which seems constant when changing
>> the size of the data set.
>
> Sorry, but I cannot reproduce this.  When using the expression:
>
> "63 + (a*b) + (c**2) + b"
>
> I get on my machine (Core2 at 3 GHz, running openSUSE Linux 11.1):
>
> 1000000 f8 (average of 10 runs)
> Simple:  0.0278068065643
> Numexpr:  0.00839750766754
> Chunked:  0.0266514062881
>
> (100, 100, 100) f8 (average of 10 runs)
> Simple:  0.0277318000793
> Numexpr:  0.00851640701294
> Chunked:  0.0346593856812
>
> and these are the expected results (i.e. no change in performance due to
> multidimensional arrays).  Even for larger arrays, I don't see nothing
> unexpected:
>
> 10000000 f8 (average of 10 runs)
> Simple:  0.334054994583
> Numexpr:  0.110022115707
> Chunked:  0.29678030014
>
> (100, 100, 100, 10) f8 (average of 10 runs)
> Simple:  0.339299607277
> Numexpr:  0.111632704735
> Chunked:  0.375299096107
>
> Can you tell us which platforms are you using?
>
>> Is this related to the way numexpr handles
>> broadcasting rules?  It would seem the memory contents should be
>> identical for these two cases.
>>
>> Andrew
>>
>> On Tue, Jan 20, 2009 at 6:13 AM, Francesc Alted <faltet at pytables.org>
> wrote:
>> > A Tuesday 20 January 2009, Andrew Collette escrigu?:
>> >> Hi Francesc,
>> >>
>> >> Looks like a cool project!  However, I'm not able to achieve the
>> >> advertised speed-ups.  I wrote a simple script to try three
>> >> approaches to this kind of problem:
>> >>
>> >> 1) Native Python code (i.e. will try to do everything at once
>> >> using temp arrays) 2) Straightforward numexpr evaluation
>> >> 3) Simple "chunked" evaluation using array.flat views.  (This
>> >> solves the memory problem and allows the use of arbitrary Python
>> >> expressions).
>> >>
>> >> I've attached the script; here's the output for the expression
>> >> "63 + (a*b) + (c**2) + sin(b)"
>> >> along with a few combinations of shapes/dtypes.  As expected,
>> >> using anything other than "f8" (double) results in a performance
>> >> penalty. Surprisingly, it seems that using chunks via array.flat
>> >> results in similar performance for f8, and even better performance
>> >> for other dtypes.
>> >
>> > [clip]
>> >
>> > Well, there were two issues there.  The first one is that when
>> > transcendental functions are used (like sin() above), the
>> > bottleneck is on the CPU instead of memory bandwidth, so numexpr
>> > speedups are not so high as usual.  The other issue was an actual
>> > bug in the numexpr code that forced a copy of all multidimensional
>> > arrays (I normally only use undimensional arrays for doing
>> > benchmarks).  This has been fixed in trunk (r39).
>> >
>> > So, with the fix on, the timings are:
>> >
>> > (100, 100, 100) f4 (average of 10 runs)
>> > Simple:  0.0426136016846
>> > Numexpr:  0.11350851059
>> > Chunked:  0.0635252952576
>> > (100, 100, 100) f8 (average of 10 runs)
>> > Simple:  0.119254398346
>> > Numexpr:  0.10092959404
>> > Chunked:  0.128384995461
>> >
>> > The speed-up is now a mere 20% (for f8), but at least it is not
>> > slower. With the patches that recently contributed Georg for using
>> > Intel's VML, the acceleration is a bit better:
>> >
>> > (100, 100, 100) f4 (average of 10 runs)
>> > Simple:  0.0417867898941
>> > Numexpr:  0.0944641113281
>> > Chunked:  0.0636183023453
>> > (100, 100, 100) f8 (average of 10 runs)
>> > Simple:  0.120059680939
>> > Numexpr:  0.0832288980484
>> > Chunked:  0.128114104271
>> >
>> > i.e. the speed-up is around 45% (for f8).
>> >
>> > Moreover, if I get rid of the sin() function and use the expresion:
>> >
>> > "63 + (a*b) + (c**2) + b"
>> >
>> > I get:
>> >
>> > (100, 100, 100) f4 (average of 10 runs)
>> > Simple:  0.0119329929352
>> > Numexpr:  0.0198570966721
>> > Chunked:  0.0338240146637
>> > (100, 100, 100) f8 (average of 10 runs)
>> > Simple:  0.0255623102188
>> > Numexpr:  0.00832500457764
>> > Chunked:  0.0340095996857
>> >
>> > which has a 3.1x speedup (for f8).
>> >
>> >> FYI, the current tar file (1.1-1) has a glitch related to the
>> >> VERSION file; I added to the bug report at google code.
>> >
>> > Thanks. Will focus on that asap.  Mmm, seems like there is stuff
>> > enough for another release of numexpr.  I'll try to do it soon.
>> >
>> > Cheers,
>> >
>> > --
>> > Francesc Alted
>> > _______________________________________________
>> > Numpy-discussion mailing list
>> > Numpy-discussion at scipy.org
>> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> --
> Francesc Alted
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From ndbecker2 at gmail.com  Wed Jan 21 13:55:49 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 13:55:49 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<200901210937.36222.lists_ravi@lavabit.com>
	<gl7ejs$do8$1@ger.gmane.org>
	<200901211130.04473.lists_ravi@lavabit.com>
Message-ID: <gl7r3l$sk1$1@ger.gmane.org>

Ravi wrote:

> On Wednesday 21 January 2009 10:22:36 Neal Becker wrote:
>> > [http://mail.python.org/pipermail/cplusplus-sig/2008-
October/013825.html
>>
>> Thanks for reminding me about this!
>>
>> Do you have a current version of the code? ?I grabbed the files from the
>> above message, but I see some additional subsequent messages with more
>> patches.
> 
> That is the latest publicly posted code. Since then, there is just one
> minor patch (attached) which enables use of row-major (c-contiguous)
> arrays.
> 
> This does *not* work with strided arrays which would be a fair bit of
> effort to support. Further, you will have to work with the numpy iterator
> interface, which, while well-designed, is a great illustration of the
> effort required to support OO programming in an non-OO language, and is
> pretty tedious to map to the ublas storage iterator interface. If you do
> implement it, I would very much like to take a look at it.
> 
> Regards,
> Ravi

I'm only interested in simple strided 1-d vectors.  In that case, I think 
your code already works.  If you have c++ code using the iterator interface, 
the iterators dereference will  use (*array )[index].  This will use 
operator[], which will call PyArray_GETPTR.  So I think this will obey 
strides.

Unfortunately, it will also be slow.  I suggest something like the enclosed.  
I have done some simple tests, and it seems to work.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy.hpp
Type: text/x-c++hdr
Size: 9909 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/8aa09eac/attachment.hpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test1.cc
Type: text/x-c++src
Size: 1359 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/8aa09eac/attachment.cc>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testsum.py
Type: text/x-python
Size: 246 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090121/8aa09eac/attachment.py>

From ndbecker2 at gmail.com  Wed Jan 21 14:57:59 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 14:57:59 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<200901210937.36222.lists_ravi@lavabit.com>
	<gl7ejs$do8$1@ger.gmane.org>
	<200901211130.04473.lists_ravi@lavabit.com>
Message-ID: <gl7uo7$9tp$1@ger.gmane.org>

Ravi wrote:

> On Wednesday 21 January 2009 10:22:36 Neal Becker wrote:
>> > [http://mail.python.org/pipermail/cplusplus-sig/2008-
October/013825.html
>>
>> Thanks for reminding me about this!
>>
>> Do you have a current version of the code? ?I grabbed the files from the
>> above message, but I see some additional subsequent messages with more
>> patches.
> 
> That is the latest publicly posted code. Since then, there is just one
> minor patch (attached) which enables use of row-major (c-contiguous)
> arrays.
> 
> This does *not* work with strided arrays which would be a fair bit of
> effort to support. Further, you will have to work with the numpy iterator
> interface, which, while well-designed, is a great illustration of the
> effort required to support OO programming in an non-OO language, and is
> pretty tedious to map to the ublas storage iterator interface. If you do
> implement it, I would very much like to take a look at it.
> 
> Regards,
> Ravi

It seems your code works fine for my usual style:

ublas::vector<T> func (numpy::array_from_py<T>::type const&)

But not for a function that modifies it arg in-place (& instead of const&):

void func (numpy::array_from_py<T>::type &)

This gives:
ArgumentError: Python argument types in
    test1.double(numpy.ndarray)
did not match C++ signature:
    double(boost::numeric::ublas::vector<std::complex<double>, 
numpy::detail::numpy_storage_array<std::complex<double> > > {lvalue})


My instinct is to ignore it, because I think I don't need it, but do you 
have a workaround?





From bpederse at gmail.com  Wed Jan 21 15:14:44 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 21 Jan 2009 12:14:44 -0800
Subject: [Numpy-discussion] genfromtxt
Message-ID: <e183a99d0901211214q6839c33as640e5f572fc72652@mail.gmail.com>

hi, i'm using the new genfromtxt stuff in numpy svn, looks great
pierre any who contributed.
is there a way to have the header commented and still be able to have
it recognized as the header? e.g.

#gender age weight
M   21  72.100000
F   35  58.330000
M   33  21.99

if i use np.loadtxt or genfromtxt, it tries to use the 2nd row (first
non-commented as the header).
and i get an array like:
array([('M', 21, 72.099999999999994), ('F', 35, 58.329999999999998),
      ('M', 33, 21.989999999999998)],
     dtype=[('f0', '|S1'), ('f1', '<i4'), ('f2', '<f8')])

when i want:

array([('M', 21, 72.099999999999994), ('F', 35, 58.329999999999998),
      ('M', 33, 21.989999999999998)],
     dtype=[('gender', '|S1'), ('age', '<i4'), ('weight', '<f8')])

i can get that by uncommenting the header, but it's useful to have
that for other code.
is there anyway to do that?
thanks,
-brent


From pgmdevlist at gmail.com  Wed Jan 21 15:28:29 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 21 Jan 2009 15:28:29 -0500
Subject: [Numpy-discussion] genfromtxt
In-Reply-To: <e183a99d0901211214q6839c33as640e5f572fc72652@mail.gmail.com>
References: <e183a99d0901211214q6839c33as640e5f572fc72652@mail.gmail.com>
Message-ID: <CBFE8FDD-B210-403F-BE5C-4F7DDF42FDDB@gmail.com>

Brent,
Currently, no, you won't be able to retrieve the header if it's  
commented.
I'll see what I can do.
P.



From lists_ravi at lavabit.com  Wed Jan 21 15:37:33 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 21 Jan 2009 15:37:33 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl7r3l$sk1$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<200901211130.04473.lists_ravi@lavabit.com>
	<gl7r3l$sk1$1@ger.gmane.org>
Message-ID: <200901211537.33966.lists_ravi@lavabit.com>

On Wednesday 21 January 2009 13:55:49 Neal Becker wrote:
> I'm only interested in simple strided 1-d vectors. ?In that case, I think
> your code already works. ?If you have c++ code using the iterator
> interface, the iterators dereference will ?use (*array )[index]. ?This will
> use operator[], which will call PyArray_GETPTR. ?So I think this will obey
> strides.

You are right. I had forgotten that I had simple strided vectors working.

> Unfortunately, it will also be slow. ?I suggest something like the
> enclosed. ? I have done some simple tests, and it seems to work.

I wonder why PyArray_GETPTR1 is slow. Is it because of the implied integer 
multiplication? Unfortunately, your approach means that iterators can become 
invalid if the underlying array is resized to a larger size. Hmmh, perhaps we 
could make this configurable at compile-time ...

Thanks for the code. Could you provide some benchmarks on the relative speeds 
of the two approaches?

Regards,
Ravi




From ndbecker2 at gmail.com  Wed Jan 21 15:53:37 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 15:53:37 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<200901211130.04473.lists_ravi@lavabit.com>
	<gl7r3l$sk1$1@ger.gmane.org>
	<200901211537.33966.lists_ravi@lavabit.com>
Message-ID: <gl820h$1t0$1@ger.gmane.org>

Ravi wrote:

> On Wednesday 21 January 2009 13:55:49 Neal Becker wrote:
>> I'm only interested in simple strided 1-d vectors. ?In that case, I think
>> your code already works. ?If you have c++ code using the iterator
>> interface, the iterators dereference will ?use (*array )[index]. ?This
>> will use operator[], which will call PyArray_GETPTR. ?So I think this
>> will obey strides.
> 
> You are right. I had forgotten that I had simple strided vectors working.
> 
>> Unfortunately, it will also be slow. ?I suggest something like the
>> enclosed. ? I have done some simple tests, and it seems to work.
> 
> I wonder why PyArray_GETPTR1 is slow. Is it because of the implied integer
> multiplication? Unfortunately, your approach means that iterators can
> become invalid if the underlying array is resized to a larger size. Hmmh,
> perhaps we could make this configurable at compile-time ...
> 
> Thanks for the code. Could you provide some benchmarks on the relative
> speeds of the two approaches?
> 
> Regards,
> Ravi
Do you know about pyublas?  This is the same issue we ran into there.  I did 
not benchmark the code you sent me.  I was just going by my experience with 
pyublas.  I guess a benchmark would be a good idea.




From lists_ravi at lavabit.com  Wed Jan 21 16:04:47 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 21 Jan 2009 16:04:47 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <gl7uo7$9tp$1@ger.gmane.org>
References: <gl604p$c9i$1@ger.gmane.org>
	<200901211130.04473.lists_ravi@lavabit.com>
	<gl7uo7$9tp$1@ger.gmane.org>
Message-ID: <200901211604.48347.lists_ravi@lavabit.com>

On Wednesday 21 January 2009 14:57:59 Neal Becker wrote:
> ublas::vector<T> func (numpy::array_from_py<T>::type const&)
>
> But not for a function that modifies it arg in-place (& instead of const&):
>
> void func (numpy::array_from_py<T>::type &)
                                          ^^^^
Use void func (numpy::array_from_py<T>::type )

Why does this work? It is a tradeoff I had to make; I chose to use python 
conventions rather than C++ conventions. Essentially, what is passed back to 
you is a reference to the numpy array. Any copies you make of it are actually 
copies of the reference, not of the actual array. This simplifies the code 
quite a bit while maintaining the reference semantics that python programmers 
use.

See dump_vec in decco.cc (the example module) for an example. 

Regards,
Ravi




From ndbecker2 at gmail.com  Wed Jan 21 16:13:59 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 16:13:59 -0500
Subject: [Numpy-discussion] line_profiler suggestion
Message-ID: <gl836p$6ee$1@ger.gmane.org>

Would be handy to not have to add/remove @profile to switch between 
profiling/normal operation.

When run without profiler, @profile is redefined to do nothing.  Possible?




From robert.kern at gmail.com  Wed Jan 21 16:34:05 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Jan 2009 15:34:05 -0600
Subject: [Numpy-discussion] line_profiler suggestion
In-Reply-To: <gl836p$6ee$1@ger.gmane.org>
References: <gl836p$6ee$1@ger.gmane.org>
Message-ID: <3d375d730901211334y39eff180s7af08dedbfc71e25@mail.gmail.com>

On Wed, Jan 21, 2009 at 15:13, Neal Becker <ndbecker2 at gmail.com> wrote:
> Would be handy to not have to add/remove @profile to switch between
> profiling/normal operation.
>
> When run without profiler, @profile is redefined to do nothing.  Possible?

I could add a --noop flag to kernprof, which basically tells it to
insert a do-nothing profile() decorator. I'm not sure what the use
case is, though. Are you switching back and forth frequently?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Wed Jan 21 16:34:35 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 16:34:35 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
References: <gl604p$c9i$1@ger.gmane.org>
	<200901211130.04473.lists_ravi@lavabit.com>
	<gl7r3l$sk1$1@ger.gmane.org>
	<200901211537.33966.lists_ravi@lavabit.com>
	<gl820h$1t0$1@ger.gmane.org>
Message-ID: <gl84dc$ans$1@ger.gmane.org>

Neal Becker wrote:

> Ravi wrote:
> 
>> On Wednesday 21 January 2009 13:55:49 Neal Becker wrote:
>>> I'm only interested in simple strided 1-d vectors. ?In that case, I
>>> think your code already works. ?If you have c++ code using the iterator
>>> interface, the iterators dereference will ?use (*array )[index]. ?This
>>> will use operator[], which will call PyArray_GETPTR. ?So I think this
>>> will obey strides.
>> 
>> You are right. I had forgotten that I had simple strided vectors working.
>> 
>>> Unfortunately, it will also be slow. ?I suggest something like the
>>> enclosed. ? I have done some simple tests, and it seems to work.
>> 
>> I wonder why PyArray_GETPTR1 is slow. Is it because of the implied
>> integer multiplication? Unfortunately, your approach means that iterators
>> can become invalid if the underlying array is resized to a larger size.
>> Hmmh, perhaps we could make this configurable at compile-time ...
>> 

Iterators almost always become invalid under those sorts of changes, so I 
don't think that's a surprise.

GETPTR1 has to do:
PyArray_STRIDES(obj)[0]
There's several memory references there, and I don't think the compiler can 
assume that this value doesn't change from one access to another - so it 
can't be cached.

That said, I have tried a few benchmarks.  Surprisingly, I'm not seeing any 
difference in a few quick tests.

I do have one cosmetic patch for you.  This will shutup gcc giving the 
longest warning message ever about an unused variable:
--- numpy.new.orig/numpyregister.hpp	2009-01-21 15:59:00.000000000 -0500
+++ numpy.new/numpyregister.hpp	2009-01-21 14:11:00.000000000 -0500
@@ -257,7 +257,8 @@
     storage_t *the_storage = reinterpret_cast<storage_t*>( data );
     void *memory_chunk = the_storage->storage.bytes;
     array_storage_t dd( obj );
-    array_t *v = new ( memory_chunk ) array_t( dd.size(), dd );
+    //array_t *v = new ( memory_chunk ) array_t( dd.size(), dd );
+    new ( memory_chunk ) array_t( dd.size(), dd );
     data->convertible = memory_chunk;
   }





From robert.kern at gmail.com  Wed Jan 21 16:36:18 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Jan 2009 15:36:18 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
Message-ID: <3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>

On Wed, Jan 21, 2009 at 12:13, Wes McKinney <wesmckinn at gmail.com> wrote:
> Robert-- this is a great little piece of code, I already think it will be a
> part of my workflow. However, I seem to be getting negative % time taken on
> the more time consuming lines, perhaps getting some overflow?

That's odd. Can you send me the code (perhaps offlist) or at least the
.lprof file?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ndbecker2 at gmail.com  Wed Jan 21 16:35:58 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 21 Jan 2009 16:35:58 -0500
Subject: [Numpy-discussion] line_profiler suggestion
References: <gl836p$6ee$1@ger.gmane.org>
	<3d375d730901211334y39eff180s7af08dedbfc71e25@mail.gmail.com>
Message-ID: <gl84fv$ans$2@ger.gmane.org>

Robert Kern wrote:

> On Wed, Jan 21, 2009 at 15:13, Neal Becker <ndbecker2 at gmail.com> wrote:
>> Would be handy to not have to add/remove @profile to switch between
>> profiling/normal operation.
>>
>> When run without profiler, @profile is redefined to do nothing. 
>> Possible?
> 
> I could add a --noop flag to kernprof, which basically tells it to
> insert a do-nothing profile() decorator. I'm not sure what the use
> case is, though. Are you switching back and forth frequently?
> 
That won't help, the idea is to not have to edit the code adding/removing 
@profile in order to run _without_ kernprof.




From robert.kern at gmail.com  Wed Jan 21 16:44:26 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Jan 2009 15:44:26 -0600
Subject: [Numpy-discussion] line_profiler suggestion
In-Reply-To: <gl84fv$ans$2@ger.gmane.org>
References: <gl836p$6ee$1@ger.gmane.org>
	<3d375d730901211334y39eff180s7af08dedbfc71e25@mail.gmail.com>
	<gl84fv$ans$2@ger.gmane.org>
Message-ID: <3d375d730901211344y59747aa9qdb189e38698a00ff@mail.gmail.com>

On Wed, Jan 21, 2009 at 15:35, Neal Becker <ndbecker2 at gmail.com> wrote:
> Robert Kern wrote:
>
>> On Wed, Jan 21, 2009 at 15:13, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> Would be handy to not have to add/remove @profile to switch between
>>> profiling/normal operation.
>>>
>>> When run without profiler, @profile is redefined to do nothing.
>>> Possible?
>>
>> I could add a --noop flag to kernprof, which basically tells it to
>> insert a do-nothing profile() decorator. I'm not sure what the use
>> case is, though. Are you switching back and forth frequently?
>>
> That won't help, the idea is to not have to edit the code adding/removing
> @profile in order to run _without_ kernprof.

In your code, you can conditionally insert a profile() decorator into
the __builtins__ if one isn't already there. You'll have to do that
before you import anything.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Jan 21 20:52:37 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Jan 2009 19:52:37 -0600
Subject: [Numpy-discussion] line_profiler suggestion
In-Reply-To: <3d375d730901211344y59747aa9qdb189e38698a00ff@mail.gmail.com>
References: <gl836p$6ee$1@ger.gmane.org>
	<3d375d730901211334y39eff180s7af08dedbfc71e25@mail.gmail.com>
	<gl84fv$ans$2@ger.gmane.org>
	<3d375d730901211344y59747aa9qdb189e38698a00ff@mail.gmail.com>
Message-ID: <3d375d730901211752n58f0846cq45dfbe08a605cf0e@mail.gmail.com>

On Wed, Jan 21, 2009 at 15:44, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Jan 21, 2009 at 15:35, Neal Becker <ndbecker2 at gmail.com> wrote:
>> Robert Kern wrote:
>>
>>> On Wed, Jan 21, 2009 at 15:13, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>> Would be handy to not have to add/remove @profile to switch between
>>>> profiling/normal operation.
>>>>
>>>> When run without profiler, @profile is redefined to do nothing.
>>>> Possible?
>>>
>>> I could add a --noop flag to kernprof, which basically tells it to
>>> insert a do-nothing profile() decorator. I'm not sure what the use
>>> case is, though. Are you switching back and forth frequently?
>>>
>> That won't help, the idea is to not have to edit the code adding/removing
>> @profile in order to run _without_ kernprof.
>
> In your code, you can conditionally insert a profile() decorator into
> the __builtins__ if one isn't already there. You'll have to do that
> before you import anything.

Alternately, you could do this in a sitecustomize.py file, but then
your code will only work on your machine. All of these workarounds are
probably little, if any, improvement over the current situation. The
answer to your original question should probably be "No".

Perhaps a better workflow for you would be the capability to specify
the desired functions in, say, a text file rather than by modifying
the code. Currently, the trace hook determines whether or not to time
a line by looking at the code object it is from. The LineProfiler has
a set of the desired code objects and tests membership by identity.
The @profile decorator populates this set by grabbing the code objects
from the function objects. This is fast and robust.

It would be possible (but difficult to make robust) to instead look at
things like .co_filename and .co_name to determine whether or not to
time a line.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Wed Jan 21 23:24:23 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 21 Jan 2009 20:24:23 -0800
Subject: [Numpy-discussion] No tests found for numpy 1.2.0
Message-ID: <6ce0ac130901212024n611ea768v84682af73565d350@mail.gmail.com>

Just installed numpy 1.2.0 and got this:

$ python -c 'import numpy; numpy.test()'
Running unit tests for numpy-1.2.0-py2.5-macosx-10.5-i386.egg.numpy

NumPy version 1.2.0
NumPy is installed in
/Users/bgranger/Library/Python/2.5/site-packages/numpy-1.2.0-py2.5-macosx-10.5-i386.egg/numpy
Python version 2.5.1 (r251:54863, Jul 23 2008, 11:00:16) [GCC 4.0.1
(Apple Inc. build 5465)]
nose version 0.10.4

----------------------------------------------------------------------
Ran 0 tests in 0.017s

OK

Any thoughts on this?  It seems to work though.  I have nose 0.10.4 installed.


From robert.kern at gmail.com  Wed Jan 21 23:36:30 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Jan 2009 22:36:30 -0600
Subject: [Numpy-discussion] No tests found for numpy 1.2.0
In-Reply-To: <6ce0ac130901212024n611ea768v84682af73565d350@mail.gmail.com>
References: <6ce0ac130901212024n611ea768v84682af73565d350@mail.gmail.com>
Message-ID: <3d375d730901212036i7d7dda05j4234b17bf9635efc@mail.gmail.com>

On Wed, Jan 21, 2009 at 22:24, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Just installed numpy 1.2.0 and got this:
>
> $ python -c 'import numpy; numpy.test()'
> Running unit tests for numpy-1.2.0-py2.5-macosx-10.5-i386.egg.numpy

easy_install sets the executable bit on files and nose ignores executable files.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Wed Jan 21 23:42:21 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 21 Jan 2009 20:42:21 -0800
Subject: [Numpy-discussion] No tests found for numpy 1.2.0
In-Reply-To: <3d375d730901212036i7d7dda05j4234b17bf9635efc@mail.gmail.com>
References: <6ce0ac130901212024n611ea768v84682af73565d350@mail.gmail.com>
	<3d375d730901212036i7d7dda05j4234b17bf9635efc@mail.gmail.com>
Message-ID: <6ce0ac130901212042m26afca11jf40faf330d68703a@mail.gmail.com>

> easy_install sets the executable bit on files and nose ignores executable files.

Thanks Robert.  I knew about this, but had never been bitten by it
yet.  Oh the joy!

Brian


From pgmdevlist at gmail.com  Thu Jan 22 00:39:21 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 22 Jan 2009 00:39:21 -0500
Subject: [Numpy-discussion] genfromtxt
In-Reply-To: <e183a99d0901211214q6839c33as640e5f572fc72652@mail.gmail.com>
References: <e183a99d0901211214q6839c33as640e5f572fc72652@mail.gmail.com>
Message-ID: <CB03EC61-B936-449C-801C-7E19C3F5DB92@gmail.com>

Brent,
Mind trying r6330 and let me know if it works for you ? Make sure that  
you use names=True to detect a header.
P.



From bpederse at gmail.com  Thu Jan 22 01:50:28 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 21 Jan 2009 22:50:28 -0800
Subject: [Numpy-discussion] genfromtxt
In-Reply-To: <CB03EC61-B936-449C-801C-7E19C3F5DB92@gmail.com>
References: <e183a99d0901211214q6839c33as640e5f572fc72652@mail.gmail.com>
	<CB03EC61-B936-449C-801C-7E19C3F5DB92@gmail.com>
Message-ID: <e183a99d0901212250r54906de4me65bc98b21c4b569@mail.gmail.com>

On Wed, Jan 21, 2009 at 9:39 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> Brent,
> Mind trying r6330 and let me know if it works for you ? Make sure that
> you use names=True to detect a header.
> P.
>

yes, works perfectly.
thanks!
-brent


From hanni.ali at gmail.com  Thu Jan 22 02:46:11 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Thu, 22 Jan 2009 07:46:11 +0000
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
	<3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
Message-ID: <789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>

I have been using your profiler extensively and it has contributed to my
achieving significant improvements in the application I work on largely due
to the usefulness of the line by line breakdown enabling me to easily select
the next part of code to work on optimizing. So firstly many thanks for
writing it.

However back to my point, Wes, I have also experienced timing oddities, in
particular on Virtual machines (MS Hyper-V has very poor processor timings,
the older MS VM works fine though). I believe the negative timings arise
when the CPU (be it virtual or possibly physical) deviates from its standard
performance or rather the initial timer unit taken, would this make sense to
you Robert?

Hanni



2009/1/21 Robert Kern <robert.kern at gmail.com>

> On Wed, Jan 21, 2009 at 12:13, Wes McKinney <wesmckinn at gmail.com> wrote:
> > Robert-- this is a great little piece of code, I already think it will be
> a
> > part of my workflow. However, I seem to be getting negative % time taken
> on
> > the more time consuming lines, perhaps getting some overflow?
>
> That's odd. Can you send me the code (perhaps offlist) or at least the
> .lprof file?
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090122/4a98993e/attachment.html>

From robert.kern at gmail.com  Thu Jan 22 02:52:32 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 22 Jan 2009 01:52:32 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
	<3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
	<789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>
Message-ID: <3d375d730901212352l112e8be8q89fd636696869d5a@mail.gmail.com>

On Thu, Jan 22, 2009 at 01:46, Hanni Ali <hanni.ali at gmail.com> wrote:
> I have been using your profiler extensively and it has contributed to my
> achieving significant improvements in the application I work on largely due
> to the usefulness of the line by line breakdown enabling me to easily select
> the next part of code to work on optimizing. So firstly many thanks for
> writing it.

My pleasure.

> However back to my point, Wes, I have also experienced timing oddities, in
> particular on Virtual machines (MS Hyper-V has very poor processor timings,
> the older MS VM works fine though). I believe the negative timings arise
> when the CPU (be it virtual or possibly physical) deviates from its standard
> performance or rather the initial timer unit taken, would this make sense to
> you Robert?

Can you try using cProfile with lots of calls to empty functions? I'm
using the same timer functions as cProfile.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Thu Jan 22 05:04:34 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 22 Jan 2009 19:04:34 +0900
Subject: [Numpy-discussion] new incremental statistics project
In-Reply-To: <200901191134.47779.meine@informatik.uni-hamburg.de>
References: <ea7d6a710812181827t34572516j981df31fbadb3f07@mail.gmail.com>
	<200901191134.47779.meine@informatik.uni-hamburg.de>
Message-ID: <5b8d13220901220204k41220b50gfbf3268865a128b@mail.gmail.com>

On Mon, Jan 19, 2009 at 7:34 PM, Hans Meine
<meine at informatik.uni-hamburg.de> wrote:
> On Friday 19 December 2008 03:27:12 Bradford Cross wrote:
>> This is a new project I just released.
>>
>> I know it is C#, but some of the design and idioms would be nice in
>> numpy/scipy for working with discrete event simulators, time series, and
>> event stream processing.
>>
>> http://code.google.com/p/incremental-statistics/
>
> Hi, do you know about the boost accumulators project?
>
> It's still in boost's sandbox, but I love its design, and it provides a large
> number of well-documented, mathematically sound estimators for variance, mean,
> etc.:
> http://boost-sandbox.sourceforge.net/libs/accumulators/doc/html/index.html
>
> Just a heads-up, in case someone finds this useful here.
> (Don't know about people's fondness of boost and/or C++ here.)

Not a boost/C++ fan, but I like those projects. Incremental statistics
have several advantages (outside the obvious one to get an online
estimate when the data arrive sequentially): they can be much more
memory friendly in a python context (for example, if you want to
compute statistics for billion of samples, you could do in mini
batches, and an incremental framework can help here), and they can
often converge faster than an offline version if you have all the
data.

I am not yet clear how pervasive those techniques are - I have looked
at several papers which prove the convergence of several well known
algorithms, and implemented some of them (in particular online EM
algorithm for online estimation of mixtures of Gaussian, with Bayesian
variations for sequential model comparison), and I would have expected
them to be more well known. I may just not be that familiar with the
concerned fields, though.

cheers,

David


From nwagner at iam.uni-stuttgart.de  Thu Jan 22 05:15:48 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 22 Jan 2009 11:15:48 +0100
Subject: [Numpy-discussion] array manipulation
Message-ID: <web-116192985@uni-stuttgart.de>

Hi all,

what is the best way to check if the entries (integers)
of an array are stored in ascending order ?

Nils
  


From cimrman3 at ntc.zcu.cz  Thu Jan 22 05:23:43 2009
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Thu, 22 Jan 2009 11:23:43 +0100
Subject: [Numpy-discussion] array manipulation
In-Reply-To: <web-116192985@uni-stuttgart.de>
References: <web-116192985@uni-stuttgart.de>
Message-ID: <4978492F.10204@ntc.zcu.cz>

Nils Wagner wrote:
> Hi all,
> 
> what is the best way to check if the entries (integers)
> of an array are stored in ascending order ?

Hi Nils,

Try np.alltrue( ar[1:] > ar[:-1] ).

r.


From nwagner at iam.uni-stuttgart.de  Thu Jan 22 05:32:28 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 22 Jan 2009 11:32:28 +0100
Subject: [Numpy-discussion] array manipulation
In-Reply-To: <4978492F.10204@ntc.zcu.cz>
References: <web-116192985@uni-stuttgart.de>
 <4978492F.10204@ntc.zcu.cz>
Message-ID: <web-116194008@uni-stuttgart.de>

On Thu, 22 Jan 2009 11:23:43 +0100
  Robert Cimrman <cimrman3 at ntc.zcu.cz> wrote:
> Nils Wagner wrote:
>> Hi all,
>> 
>> what is the best way to check if the entries (integers)
>> of an array are stored in ascending order ?
> 
> Hi Nils,
> 
> Try np.alltrue( ar[1:] > ar[:-1] ).
> 
> r.
  
Thank you !

Nils


From nwagner at iam.uni-stuttgart.de  Thu Jan 22 06:49:48 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 22 Jan 2009 12:49:48 +0100
Subject: [Numpy-discussion] header option for savetxt
Message-ID: <web-116197593@uni-stuttgart.de>

Hi all,

Is it possible to add a header option for savetxt ?
  
Nils


From chris at simplistix.co.uk  Thu Jan 22 07:02:01 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 22 Jan 2009 12:02:01 +0000
Subject: [Numpy-discussion] Tutorial on working with Excel files in Python
 (without COM and cross platform!) at PyConUS 2009
Message-ID: <49786039.2020101@simplistix.co.uk>

Hi All,

Too many people in the Python community think the only way to work with
Excel files in Python is using COM on Windows.

To try and correct this, I'm giving a tutorial at this year's PyCon in 
Chicago on Wednesday, 25th March that will cover working with Excel 
files in Python using the pure-python libraries xlrd, xlwt and xlutils.

I'll be looking to cover:

- Reading Excel Files

   Including formatting, unicode dates and formulae.

- Writing Excel Files

   Including formatting with easyxf and things like freeze pains, print
   areas, etc

- Filtering Excel Files

   A run through on the structure of xlutils.filter and some examples to
   show you how it works.

- Workshop for your problems

   I'm hoping anyone who attends will get a lot out of this! If you're
   planning on attending and have a particular problem you'd like to work
   on in this part of the tutorial, please drop me an email and I'll try
   and make sure I come prepared!

All you need for the tutorial is a working knowledge of Excel and 
Python, with a laptop as an added benefit, and to be at PyCon this year:

http://us.pycon.org

I look forward to seeing you all there!

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk



From dsdale24 at gmail.com  Thu Jan 22 09:15:37 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Thu, 22 Jan 2009 09:15:37 -0500
Subject: [Numpy-discussion] numpy.array and subok kwarg
Message-ID: <a08e5f80901220615r654cacfalb9ba6b60cf2cc6ee@mail.gmail.com>

Hello,

I have a test script:


import numpy as np

class MyArray(np.ndarray):

    __array_priority__ = 20

    def __new__(cls):
        return np.asarray(1).view(cls).copy()

    def __repr__(self):
        return 'my_array'

    __str__ = __repr__

    def __mul__(self, other):
        return super(MyArray, self).__mul__(other)

    def __rmul__(self, other):
        return super(MyArray, self).__rmul__(other)

mine = MyArray()
print type(np.array(mine,dtype='f'))



The type returned by np.array is ndarray, unless I specifically set
subok=True, in which case I get a MyArray. The default value of subok is
True, so I dont understand why I have to specify subok unless I want it to
be False. Is my subclass missing something important?

Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090122/3e2ac281/attachment.html>

From pgmdevlist at gmail.com  Thu Jan 22 10:53:45 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 22 Jan 2009 10:53:45 -0500
Subject: [Numpy-discussion] numpy.array and subok kwarg
In-Reply-To: <a08e5f80901220615r654cacfalb9ba6b60cf2cc6ee@mail.gmail.com>
References: <a08e5f80901220615r654cacfalb9ba6b60cf2cc6ee@mail.gmail.com>
Message-ID: <29F30CE2-DED7-4A3C-8C1F-3057205E0A74@gmail.com>

Darren,


> The type returned by np.array is ndarray, unless I specifically set  
> subok=True, in which case I get a MyArray. The default value of  
> subok is True, so I dont understand why I have to specify subok  
> unless I want it to be False. Is my subclass missing something  
> important?

Blame the doc: the default for subok in array is False, as explicit in  
the _array_fromobject Cfunction (in multiarray). So no, you're not  
doing anything wrong. Note that by default subok=True for  
numpy.ma.array.


From reakinator at gmail.com  Thu Jan 22 11:45:51 2009
From: reakinator at gmail.com (Rich E)
Date: Thu, 22 Jan 2009 17:45:51 +0100
Subject: [Numpy-discussion] default float type of array not accepted by SWIG
	wrapped C functions
Message-ID: <a3297bc90901220845y72907465q3800384d212ab9b8@mail.gmail.com>

Hi all,

I have a SWIG wrapped C library that uses 32bit floating point arrays,
using the numpy.i typemapping system for passing the arrays.  For
every array that I make, I have to convert it using astype('float32'),
else python complains that I tried to pass a double-precision array.

Is there any way to set the default floating point precision to 32bit,
in python or in the SWIG interface file?

regards,
Rich


From wfspotz at sandia.gov  Thu Jan 22 12:09:01 2009
From: wfspotz at sandia.gov (Spotz, William F)
Date: Thu, 22 Jan 2009 10:09:01 -0700
Subject: [Numpy-discussion] default float type of array not accepted by
 SWIG wrapped C functions
In-Reply-To: <a3297bc90901220845y72907465q3800384d212ab9b8@mail.gmail.com>
References: <a3297bc90901220845y72907465q3800384d212ab9b8@mail.gmail.com>
Message-ID: <875E04698300DB4FA52B4219ABA6039B126E9D05BD@ES02SNLNT.srn.sandia.gov>

Rich,

Basic python only supports double precision floats, so that is not an option.

NumPy does not have, as far as I know, a way to set the default precision, although it might be a reasonable request.

As for the SWIG interface file, almost anything is possible.  Can you give an example of a function prototype you are wrapping, the %apply directive you use and and example of python code accessing it?

-Bill
________________________________________
From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Rich E [reakinator at gmail.com]
Sent: Thursday, January 22, 2009 11:45 AM
To: Discussion of Numerical Python
Subject: [Numpy-discussion] default float type of array not accepted by SWIG wrapped C functions

Hi all,

I have a SWIG wrapped C library that uses 32bit floating point arrays,
using the numpy.i typemapping system for passing the arrays.  For
every array that I make, I have to convert it using astype('float32'),
else python complains that I tried to pass a double-precision array.

Is there any way to set the default floating point precision to 32bit,
in python or in the SWIG interface file?

regards,
Rich
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From nwagner at iam.uni-stuttgart.de  Thu Jan 22 12:27:47 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 22 Jan 2009 18:27:47 +0100
Subject: [Numpy-discussion] Failures in test_recfunctions.py
Message-ID: <web-116215307@uni-stuttgart.de>

>>> numpy.__version__
'1.3.0.dev6331'

======================================================================
FAIL: Test the ignoremask option of find_duplicates
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", 
line 186, in test_find_duplicates_ignoremask
     assert_equal(test[-1], control)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", 
line 121, in assert_equal
     return assert_array_equal(actual, desired, err_msg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", 
line 193, in assert_array_equal
     header='Arrays are not equal')
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", 
line 186, in assert_array_compare
     verbose=verbose, header=header)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", 
line 295, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 33.3333333333%)
  x: array([0, 1, 3, 4, 2, 6])
  y: array([0, 1, 3, 4, 6, 2])

======================================================================
FAIL: Test zip_descr
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", 
line 34, in test_zip_descr
     np.dtype([('', '<i4'), ('', '<i4')]))
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", 
line 98, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal:
  ACTUAL: [('f0', '<i8'), ('f1', '<i8')]
  DESIRED: dtype([('f0', '<i4'), ('f1', '<i4')])

----------------------------------------------------------------------
Ran 1881 tests in 15.223s

FAILED (KNOWNFAIL=9, failures=2)
<nose.result.TextTestResult run=1881 errors=0 failures=2>


From pgmdevlist at gmail.com  Thu Jan 22 12:53:39 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 22 Jan 2009 12:53:39 -0500
Subject: [Numpy-discussion] Failures in test_recfunctions.py
In-Reply-To: <web-116215307@uni-stuttgart.de>
References: <web-116215307@uni-stuttgart.de>
Message-ID: <56A4AC64-CDD0-425E-83BB-F9820212142A@gmail.com>

Interesting.
The tests pass on my machine
OS X,
Python version 2.5.4 (r254:67916, Dec 29 2008, 17:02:44) [GCC 4.0.1  
(Apple Inc. build 5488)]
nose version 0.10.4

For
>   File
> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/ 
> test_recfunctions.py",
> line 34, in test_zip_descr
>     np.dtype([('', '<i4'), ('', '<i4')]))

I guess I can change '<i4' to int, which should work.

For:
>
> ======================================================================
> FAIL: Test the ignoremask option of find_duplicates
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/ 
> test_recfunctions.py",
> line 186, in test_find_duplicates_ignoremask
>     assert_equal(test[-1], control)
>
> (mismatch 33.3333333333%)
>  x: array([0, 1, 3, 4, 2, 6])
>  y: array([0, 1, 3, 4, 6, 2])
>

there's obviously a machine-dependent element somewhere. I'd blame  
argsort: the last 2 indices that are switched correspond to the masked  
elements in the input of the test. Note that the result is basically  
correct.
I should have access to a linux box, I'll see what I can do.




From nwagner at iam.uni-stuttgart.de  Thu Jan 22 13:31:09 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 22 Jan 2009 19:31:09 +0100
Subject: [Numpy-discussion] Failures in test_recfunctions.py
In-Reply-To: <56A4AC64-CDD0-425E-83BB-F9820212142A@gmail.com>
References: <web-116215307@uni-stuttgart.de>
	<56A4AC64-CDD0-425E-83BB-F9820212142A@gmail.com>
Message-ID: <web-116217089@uni-stuttgart.de>

On Thu, 22 Jan 2009 12:53:39 -0500
  Pierre GM <pgmdevlist at gmail.com> wrote:
> Interesting.
> The tests pass on my machine
> OS X,
> Python version 2.5.4 (r254:67916, Dec 29 2008, 17:02:44) 
>[GCC 4.0.1  
> (Apple Inc. build 5488)]
> nose version 0.10.4
> 
>For
>>   File
>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/ 
>> test_recfunctions.py",
>> line 34, in test_zip_descr
>>     np.dtype([('', '<i4'), ('', '<i4')]))
> 
> I guess I can change '<i4' to int, which should work.
> 
>For:
>>
>> ======================================================================
>> FAIL: Test the ignoremask option of find_duplicates
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File
>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/ 
>> test_recfunctions.py",
>> line 186, in test_find_duplicates_ignoremask
>>     assert_equal(test[-1], control)
>>
>> (mismatch 33.3333333333%)
>>  x: array([0, 1, 3, 4, 2, 6])
>>  y: array([0, 1, 3, 4, 6, 2])
>>
> 
> there's obviously a machine-dependent element somewhere. 
>I'd blame  
> argsort: the last 2 indices that are switched correspond 
>to the masked  
> elements in the input of the test. Note that the result 
>is basically  
> correct.
> I should have access to a linux box, I'll see what I can 
>do.
> 
Hi Pierre,

Thank you. Works for me.

Ran 1881 tests in 12.565s

OK (KNOWNFAIL=9)
<nose.result.TextTestResult run=1881 errors=0 failures=0>
  
Nils


From pgmdevlist at gmail.com  Thu Jan 22 13:40:32 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 22 Jan 2009 13:40:32 -0500
Subject: [Numpy-discussion] Failures in test_recfunctions.py
In-Reply-To: <web-116217089@uni-stuttgart.de>
References: <web-116215307@uni-stuttgart.de>
	<56A4AC64-CDD0-425E-83BB-F9820212142A@gmail.com>
	<web-116217089@uni-stuttgart.de>
Message-ID: <9F1B944B-A9B8-410F-A3A5-DFA66ED34099@gmail.com>


On Jan 22, 2009, at 1:31 PM, Nils Wagner wrote:
>>
> Hi Pierre,
>
> Thank you. Works for me.

You're welcome, thanks for reporting!


From haase at msg.ucsf.edu  Thu Jan 22 16:37:45 2009
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Thu, 22 Jan 2009 22:37:45 +0100
Subject: [Numpy-discussion] have to use matlab: what is the equiv. to
	pdb.pm()
Message-ID: <bc657ead0901221337i538c6ddeu569667adca36f7fc@mail.gmail.com>

Hi,
I have to debug a matlab program -- is there a way to do a "post
mortem" debugging ?
(I don't even know if this term even exists outside of python ;-)
((I want to know some local variable values and find out why our
program crashes when "there are no particles found"...))


Thanks,
Sebastian Haase

PS: This is obviously somewhat OT


From matthew.brett at gmail.com  Thu Jan 22 17:47:43 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 22 Jan 2009 14:47:43 -0800
Subject: [Numpy-discussion] have to use matlab: what is the equiv. to
	pdb.pm()
In-Reply-To: <bc657ead0901221337i538c6ddeu569667adca36f7fc@mail.gmail.com>
References: <bc657ead0901221337i538c6ddeu569667adca36f7fc@mail.gmail.com>
Message-ID: <1e2af89e0901221447o157d1490ka3eac7d418cb3039@mail.gmail.com>

Did you say "matlab" ?  ;-)

> I have to debug a matlab program -- is there a way to do a "post
> mortem" debugging ?
> (I don't even know if this term even exists outside of python ;-)
> ((I want to know some local variable values and find out why our
> program crashes when "there are no particles found"...))

I think you want:

>> dbstop if error
>> my_script

Best,

Matthew


From wesmckinn at gmail.com  Thu Jan 22 18:00:52 2009
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 22 Jan 2009 18:00:52 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <3d375d730901212352l112e8be8q89fd636696869d5a@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
	<3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
	<789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>
	<3d375d730901212352l112e8be8q89fd636696869d5a@mail.gmail.com>
Message-ID: <6c476c8a0901221500h4b950f75ob1508ef6fbcf56c5@mail.gmail.com>

import cProfile

def f():
    pass

def g():
    for i in xrange(1000000):
        f()

cProfile.run("g()")

>test.py
         1000003 function calls in 1.225 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.225    1.225 <string>:1(<module>)
  1000000    0.464    0.000    0.464    0.000 test.py:3(f)
        1    0.761    0.761    1.225    1.225 test.py:6(g)
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}

Running this with line_profiler:

Timer unit: 2.9485e-010 s

File: test.py
Function: g at line 9
Total time: 0.855075 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     9                                           @profiler
    10                                           def g():
    11   1000001   1844697930   1844.7     63.6         for i in
xrange(1000000):
    12   1000000   1055333053   1055.3     36.4                 f()

Which is what I would expect. Hmm

On Thu, Jan 22, 2009 at 2:52 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Thu, Jan 22, 2009 at 01:46, Hanni Ali <hanni.ali at gmail.com> wrote:
> > I have been using your profiler extensively and it has contributed to my
> > achieving significant improvements in the application I work on largely
> due
> > to the usefulness of the line by line breakdown enabling me to easily
> select
> > the next part of code to work on optimizing. So firstly many thanks for
> > writing it.
>
> My pleasure.
>
> > However back to my point, Wes, I have also experienced timing oddities,
> in
> > particular on Virtual machines (MS Hyper-V has very poor processor
> timings,
> > the older MS VM works fine though). I believe the negative timings arise
> > when the CPU (be it virtual or possibly physical) deviates from its
> standard
> > performance or rather the initial timer unit taken, would this make sense
> to
> > you Robert?
>
> Can you try using cProfile with lots of calls to empty functions? I'm
> using the same timer functions as cProfile.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090122/9daddaf0/attachment.html>

From robert.kern at gmail.com  Thu Jan 22 18:03:35 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 22 Jan 2009 17:03:35 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <6c476c8a0901221500h4b950f75ob1508ef6fbcf56c5@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
	<3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
	<789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>
	<3d375d730901212352l112e8be8q89fd636696869d5a@mail.gmail.com>
	<6c476c8a0901221500h4b950f75ob1508ef6fbcf56c5@mail.gmail.com>
Message-ID: <3d375d730901221503ufd3fc6eu513c9e82d068c36a@mail.gmail.com>

On Thu, Jan 22, 2009 at 17:00, Wes McKinney <wesmckinn at gmail.com> wrote:
> import cProfile
>
> def f():
>     pass
>
> def g():
>     for i in xrange(1000000):
>         f()
>
> cProfile.run("g()")
>
>>test.py
>          1000003 function calls in 1.225 CPU seconds
>
>    Ordered by: standard name
>
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>         1    0.000    0.000    1.225    1.225 <string>:1(<module>)
>   1000000    0.464    0.000    0.464    0.000 test.py:3(f)
>         1    0.761    0.761    1.225    1.225 test.py:6(g)
>         1    0.000    0.000    0.000    0.000 {method 'disable' of
> '_lsprof.Profiler' objects}
>
> Running this with line_profiler:
>
> Timer unit: 2.9485e-010 s
>
> File: test.py
> Function: g at line 9
> Total time: 0.855075 s
>
> Line #      Hits         Time  Per Hit   % Time  Line Contents
> ==============================================================
>      9                                           @profiler
>     10                                           def g():
>     11   1000001   1844697930   1844.7     63.6         for i in
> xrange(1000000):
>     12   1000000   1055333053   1055.3     36.4                 f()
>
> Which is what I would expect. Hmm

What platform are you on?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From wesmckinn at gmail.com  Thu Jan 22 18:09:05 2009
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 22 Jan 2009 18:09:05 -0500
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <3d375d730901221503ufd3fc6eu513c9e82d068c36a@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org>
	<3d375d730901201822g5a7e5e9fi3d341502b0d59878@mail.gmail.com>
	<gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
	<3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
	<789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>
	<3d375d730901212352l112e8be8q89fd636696869d5a@mail.gmail.com>
	<6c476c8a0901221500h4b950f75ob1508ef6fbcf56c5@mail.gmail.com>
	<3d375d730901221503ufd3fc6eu513c9e82d068c36a@mail.gmail.com>
Message-ID: <6c476c8a0901221509w7f454840j51eeec98fdac4c16@mail.gmail.com>

Windows XP, Pentium D, Python 2.5.2

On Thu, Jan 22, 2009 at 6:03 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Thu, Jan 22, 2009 at 17:00, Wes McKinney <wesmckinn at gmail.com> wrote:
> > import cProfile
> >
> > def f():
> >     pass
> >
> > def g():
> >     for i in xrange(1000000):
> >         f()
> >
> > cProfile.run("g()")
> >
> >>test.py
> >          1000003 function calls in 1.225 CPU seconds
> >
> >    Ordered by: standard name
> >
> >    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> >         1    0.000    0.000    1.225    1.225 <string>:1(<module>)
> >   1000000    0.464    0.000    0.464    0.000 test.py:3(f)
> >         1    0.761    0.761    1.225    1.225 test.py:6(g)
> >         1    0.000    0.000    0.000    0.000 {method 'disable' of
> > '_lsprof.Profiler' objects}
> >
> > Running this with line_profiler:
> >
> > Timer unit: 2.9485e-010 s
> >
> > File: test.py
> > Function: g at line 9
> > Total time: 0.855075 s
> >
> > Line #      Hits         Time  Per Hit   % Time  Line Contents
> > ==============================================================
> >      9                                           @profiler
> >     10                                           def g():
> >     11   1000001   1844697930   1844.7     63.6         for i in
> > xrange(1000000):
> >     12   1000000   1055333053   1055.3     36.4                 f()
> >
> > Which is what I would expect. Hmm
>
> What platform are you on?
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090122/b47b7944/attachment.html>

From rw247 at astro.columbia.edu  Thu Jan 22 19:28:37 2009
From: rw247 at astro.columbia.edu (Ross Williamson)
Date: Fri, 23 Jan 2009 13:28:37 +1300
Subject: [Numpy-discussion] range in numpy where
Message-ID: <6EE54750-9AEB-40A3-AD35-D2C1063E0770@astro.columbia.edu>

Hi All

I want to get out the index of values in an array. Normally WHERE  
works fine for one conditional statement but it does not work for two  
- i.e.

a = array([0,1,2,3,4,5,6,7,8,9])

ind, = where(a > 5)

Works fine but if I wanted:

ind = where((a > 5) and (a<8))

Then it bugs out with the following message:

"The truth value of an array with more than one message is ambiguous"

Any ideas?

Cheers

Ross


From efiring at hawaii.edu  Thu Jan 22 19:57:02 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Thu, 22 Jan 2009 14:57:02 -1000
Subject: [Numpy-discussion] range in numpy where
In-Reply-To: <6EE54750-9AEB-40A3-AD35-D2C1063E0770@astro.columbia.edu>
References: <6EE54750-9AEB-40A3-AD35-D2C1063E0770@astro.columbia.edu>
Message-ID: <497915DE.809@hawaii.edu>

Ross Williamson wrote:
> Hi All
> 
> I want to get out the index of values in an array. Normally WHERE  
> works fine for one conditional statement but it does not work for two  
> - i.e.
> 
> a = array([0,1,2,3,4,5,6,7,8,9])
> 
> ind, = where(a > 5)
> 
> Works fine but if I wanted:
> 
> ind = where((a > 5) and (a<8))

Python/numpy wart: for fairly fundamental Python reasons, "and" does not 
work elementwise so you have to use the "&" operator instead:

ind = where((a > 5) & (a<8))

Same for "or" and "|".

Parentheses as you have them are essential because of the precedence of 
the bit operators.

Eric
> 
> Then it bugs out with the following message:
> 
> "The truth value of an array with more than one message is ambiguous"
> 
> Any ideas?
> 
> Cheers
> 
> Ross
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From strawman at astraw.com  Thu Jan 22 20:38:41 2009
From: strawman at astraw.com (Andrew Straw)
Date: Thu, 22 Jan 2009 17:38:41 -0800
Subject: [Numpy-discussion] N-D array interface page is out of date
Message-ID: <49791FA1.3020803@astraw.com>

Hi, I just noticed that the N-D array interface page is outdated and
doesn't mention the buffer interface that is standard with Python 2.6
and Python 3.0:

http://numpy.scipy.org/array_interface.shtml

This page is linked to from http://numpy.scipy.org/

I suggest, at the minimum, modifying the page with really annoying
blinking red letters at the top (or other suitable warning) that this is
deprecated at that people should use
http://www.python.org/dev/peps/pep-3118/ instead.


From scott.sinclair.za at gmail.com  Fri Jan 23 00:44:11 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Fri, 23 Jan 2009 07:44:11 +0200
Subject: [Numpy-discussion] numpy.array and subok kwarg
In-Reply-To: <29F30CE2-DED7-4A3C-8C1F-3057205E0A74@gmail.com>
References: <a08e5f80901220615r654cacfalb9ba6b60cf2cc6ee@mail.gmail.com>
	<29F30CE2-DED7-4A3C-8C1F-3057205E0A74@gmail.com>
Message-ID: <6a17e9ee0901222144k4dc4f4c5o38d883602d85218@mail.gmail.com>

> 2009/1/22 Pierre GM <pgmdevlist at gmail.com>:
> Darren,
>
>
>> The type returned by np.array is ndarray, unless I specifically set
>> subok=True, in which case I get a MyArray. The default value of
>> subok is True, so I dont understand why I have to specify subok
>> unless I want it to be False. Is my subclass missing something
>> important?
>
> Blame the doc: the default for subok in array is False, as explicit in
> the _array_fromobject Cfunction (in multiarray). So no, you're not
> doing anything wrong. Note that by default subok=True for
> numpy.ma.array.

Corrected in the doc-editor.

Cheers,
Scott


From sole at esrf.fr  Fri Jan 23 02:11:21 2009
From: sole at esrf.fr (V. Armando Sole)
Date: Fri, 23 Jan 2009 08:11:21 +0100
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
Message-ID: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>

Hello,

In an effort to suppress for loops, I have arrived to the following situation.

Through vectorial logical operations I generate a set of indices for which 
the contents of an array have to be incremented. My problem can be reduced 
to the following:

#This works
import numpy
a=numpy.zeros(10)
b=numpy.ones(4, numpy.int)

for i in b:
     a[i] += 1
#a[1] contains 4 at the end


#This does not work
import numpy
a=numpy.zeros(10)
b=numpy.ones(4, numpy.int)
a[b] += 1

#a[1] contains 1 at the end

Is that a bug or a feature? Is there a way I can achieve the first result 
without a for loop? In my application the difference is a factor 10 in 
execution time (1000 secons instead of 100 ...)

Thanks,

Armando




From matthew.brett at gmail.com  Fri Jan 23 02:39:16 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 23 Jan 2009 02:39:16 -0500
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
In-Reply-To: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
References: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
Message-ID: <1e2af89e0901222339y75526929tdb63c0c4fb18484@mail.gmail.com>

Hi,

> #This does not work
> import numpy
> a=numpy.zeros(10)
> b=numpy.ones(4, numpy.int)
> a[b] += 1

The problem here is that you are setting a[1]  to a[1]+1.

I think you want:

import numpy
a=numpy.zeros(10)
b=numpy.ones(4, numpy.bool)
a[b] += 1

Best,

Matthew


From robert.kern at gmail.com  Fri Jan 23 02:42:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 23 Jan 2009 01:42:40 -0600
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
In-Reply-To: <1e2af89e0901222339y75526929tdb63c0c4fb18484@mail.gmail.com>
References: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
	<1e2af89e0901222339y75526929tdb63c0c4fb18484@mail.gmail.com>
Message-ID: <3d375d730901222342p4ef7d902hc747b2a587d5a335@mail.gmail.com>

On Fri, Jan 23, 2009 at 01:39, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
>> #This does not work
>> import numpy
>> a=numpy.zeros(10)
>> b=numpy.ones(4, numpy.int)
>> a[b] += 1
>
> The problem here is that you are setting a[1]  to a[1]+1.
>
> I think you want:
>
> import numpy
> a=numpy.zeros(10)
> b=numpy.ones(4, numpy.bool)
> a[b] += 1

Judging from his for loop, he does want the integer array. He's doing
something like histogramming.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan 23 02:44:14 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 23 Jan 2009 01:44:14 -0600
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
In-Reply-To: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
References: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
Message-ID: <3d375d730901222344g3c7274fdg183c8356885915b1@mail.gmail.com>

On Fri, Jan 23, 2009 at 01:11, V. Armando Sole <sole at esrf.fr> wrote:
> Hello,
>
> In an effort to suppress for loops, I have arrived to the following situation.
>
> Through vectorial logical operations I generate a set of indices for which
> the contents of an array have to be incremented. My problem can be reduced
> to the following:
>
> #This works
> import numpy
> a=numpy.zeros(10)
> b=numpy.ones(4, numpy.int)
>
> for i in b:
>     a[i] += 1
> #a[1] contains 4 at the end
>
>
> #This does not work
> import numpy
> a=numpy.zeros(10)
> b=numpy.ones(4, numpy.int)
> a[b] += 1
>
> #a[1] contains 1 at the end
>
> Is that a bug or a feature?

It is an inevitable consequence of several features interacting
together. Basically, Python expands "a[b] += 1" into this:

  c = a[b]
  d = c.__iadd__(1)
  a[b] = d

Basically, the array c doesn't know that it was created by indexing a,
so it can't do the accumulation you want.

> Is there a way I can achieve the first result
> without a for loop? In my application the difference is a factor 10 in
> execution time (1000 secons instead of 100 ...)

In [6]: bincount?
Type:           builtin_function_or_method
Base Class:     <type 'builtin_function_or_method'>
String Form:    <built-in function bincount>
Namespace:      Interactive
Docstring:
    bincount(x,weights=None)

    Return the number of occurrences of each value in x.

    x must be a list of non-negative integers.  The output, b[i],
    represents the number of times that i is found in x.  If weights
    is specified, every occurrence of i at a position p contributes
    weights[p] instead of 1.

    See also: histogram, digitize, unique.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From matthew.brett at gmail.com  Fri Jan 23 02:52:40 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 22 Jan 2009 23:52:40 -0800
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
In-Reply-To: <3d375d730901222342p4ef7d902hc747b2a587d5a335@mail.gmail.com>
References: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
	<1e2af89e0901222339y75526929tdb63c0c4fb18484@mail.gmail.com>
	<3d375d730901222342p4ef7d902hc747b2a587d5a335@mail.gmail.com>
Message-ID: <1e2af89e0901222352i1f447a97r692aeed3ff2f1250@mail.gmail.com>

> Judging from his for loop, he does want the integer array. He's doing
> something like histogramming.

Yup, thanks, just goes to show that it's not good to send emails after
a glass of wine late at night with slight jetlag.

Matthew


From sole at esrf.fr  Fri Jan 23 03:10:41 2009
From: sole at esrf.fr (V. Armando Sole)
Date: Fri, 23 Jan 2009 09:10:41 +0100
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
In-Reply-To: <3d375d730901222344g3c7274fdg183c8356885915b1@mail.gmail.co
 m>
References: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
	<5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
Message-ID: <5.2.0.9.1.20090123085405.033d3878@pop.esrf.fr>

At 01:44 23/01/2009 -0600, Robert Kern wrote:
>It is an inevitable consequence of several features interacting
>together. Basically, Python expands "a[b] += 1" into this:
>
>   c = a[b]
>   d = c.__iadd__(1)
>   a[b] = d
>
>Basically, the array c doesn't know that it was created by indexing a,
>so it can't do the accumulation you want.

Well inevitable would not be the word I would use. I would have expected a 
different behaviour between a[b] = a[b] + 1 and a[b] += 1

In the second case python (or numpy) does not need to generate an 
intermediate array and could be doing in-place operations.


> > Is there a way I can achieve the first result
> > without a for loop? In my application the difference is a factor 10 in
> > execution time (1000 secons instead of 100 ...)
>
>In [6]: bincount?
>Type:           builtin_function_or_method
>Base Class:     <type 'builtin_function_or_method'>
>String Form:    <built-in function bincount>
>Namespace:      Interactive
>Docstring:
>     bincount(x,weights=None)
>
>     Return the number of occurrences of each value in x.
>
>     x must be a list of non-negative integers.  The output, b[i],
>     represents the number of times that i is found in x.  If weights
>     is specified, every occurrence of i at a position p contributes
>     weights[p] instead of 1.
>
>     See also: histogram, digitize, unique.

Indeed what I am doing is very close to histogramming.

Unfortunaly what I have to add is not just one but a value. I have a set of 
scattered points (x,y,z) and values corresponding to those points. My goal 
is to get a regular grid in which in each voxel I sum the values of the 
points falling in them. I guess I will have to write a tiny C extension. I 
had expected the += sintax would trigger the in-place operation.

Best regards,

Armando 




From robert.kern at gmail.com  Fri Jan 23 03:17:03 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 23 Jan 2009 02:17:03 -0600
Subject: [Numpy-discussion] Numpy unexpected (for me) behaviour
In-Reply-To: <5.2.0.9.1.20090123085405.033d3878@pop.esrf.fr>
References: <5.2.0.9.1.20090123075859.03438b10@pop.esrf.fr>
	<5.2.0.9.1.20090123085405.033d3878@pop.esrf.fr>
Message-ID: <3d375d730901230017g36aca5bdx6cfdf6fe504df17e@mail.gmail.com>

On Fri, Jan 23, 2009 at 02:10, V. Armando Sole <sole at esrf.fr> wrote:
> At 01:44 23/01/2009 -0600, Robert Kern wrote:
>>It is an inevitable consequence of several features interacting
>>together. Basically, Python expands "a[b] += 1" into this:
>>
>>   c = a[b]
>>   d = c.__iadd__(1)
>>   a[b] = d
>>
>>Basically, the array c doesn't know that it was created by indexing a,
>>so it can't do the accumulation you want.
>
> Well inevitable would not be the word I would use. I would have expected a
> different behaviour between a[b] = a[b] + 1 and a[b] += 1
>
> In the second case python (or numpy) does not need to generate an
> intermediate array and could be doing in-place operations.

I'm sorry, but that is how Python implements a[b] += 1. We have no
control over it. There is simply no hook to implement that statement
as an atomic unit. Instead, we have three hooks, __getitem__,
__setitem__, and __iadd__. __iadd__ is orthogonal to the former two,
so it can't know that it is operating in that context.

>> > Is there a way I can achieve the first result
>> > without a for loop? In my application the difference is a factor 10 in
>> > execution time (1000 secons instead of 100 ...)
>>
>>In [6]: bincount?
>>Type:           builtin_function_or_method
>>Base Class:     <type 'builtin_function_or_method'>
>>String Form:    <built-in function bincount>
>>Namespace:      Interactive
>>Docstring:
>>     bincount(x,weights=None)
>>
>>     Return the number of occurrences of each value in x.
>>
>>     x must be a list of non-negative integers.  The output, b[i],
>>     represents the number of times that i is found in x.  If weights
>>     is specified, every occurrence of i at a position p contributes
>>     weights[p] instead of 1.
>>
>>     See also: histogram, digitize, unique.
>
> Indeed what I am doing is very close to histogramming.
>
> Unfortunaly what I have to add is not just one but a value. I have a set of
> scattered points (x,y,z) and values corresponding to those points. My goal
> is to get a regular grid in which in each voxel I sum the values of the
> points falling in them.

That's what the weights input is for.

> I guess I will have to write a tiny C extension. I
> had expected the += sintax would trigger the in-place operation.

It does, just a different one than you want.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan 23 03:19:59 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 23 Jan 2009 02:19:59 -0600
Subject: [Numpy-discussion] python numpy code many times slower than c++
In-Reply-To: <6c476c8a0901221509w7f454840j51eeec98fdac4c16@mail.gmail.com>
References: <gl604p$c9i$1@ger.gmane.org> <gl62ua$hpt$1@ger.gmane.org>
	<3d375d730901210023u56a260a9k24646e3fb9f41d70@mail.gmail.com>
	<6c476c8a0901211013k42858881rc9b3703a341b415f@mail.gmail.com>
	<3d375d730901211336l55d13ffev35f6980cef57be35@mail.gmail.com>
	<789d27b10901212346g2a23c28kebf0af9f229f5840@mail.gmail.com>
	<3d375d730901212352l112e8be8q89fd636696869d5a@mail.gmail.com>
	<6c476c8a0901221500h4b950f75ob1508ef6fbcf56c5@mail.gmail.com>
	<3d375d730901221503ufd3fc6eu513c9e82d068c36a@mail.gmail.com>
	<6c476c8a0901221509w7f454840j51eeec98fdac4c16@mail.gmail.com>
Message-ID: <3d375d730901230019uf982fedr1ce359de8a389db5@mail.gmail.com>

On Thu, Jan 22, 2009 at 17:09, Wes McKinney <wesmckinn at gmail.com> wrote:
> Windows XP, Pentium D, Python 2.5.2

I can replicate the negative numbers on my Windows VM. I'll take a look at it.

Wrote profile results to foo.py.lprof
Timer unit: 4.17601e-010 s

File: foo.py
Function: f at line 1
Total time: -3.02963 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile
     2                                           def f():
     3   1000001  -1456737621  -1456.7     20.1         for i in
xrange(1000000):
     4   1000000  -1540435131  -1540.4     21.2                 1+1
     5   1000000  -1522306067  -1522.3     21.0                 1+1
     6   1000000  -1177199444  -1177.2     16.2                 1+1
     7   1000000  -1558164209  -1558.2     21.5                 1+1


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From timmichelsen at gmx-topmail.de  Thu Jan 22 11:10:33 2009
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Thu, 22 Jan 2009 17:10:33 +0100
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <496A5CC0.8050605@american.edu>
References: <496A5CC0.8050605@american.edu>
Message-ID: <glcc5i$qsh$1@ger.gmane.org>

Hello Allan, Stefan and others,
did you already come to a conclusion regarding this cite topic?

Did you try to run the bibtext extension for Sphinx?

If so, please update the documentation guidelines.

Regards,
Timmie



From stefan at sun.ac.za  Fri Jan 23 07:39:05 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 23 Jan 2009 14:39:05 +0200
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <glcc5i$qsh$1@ger.gmane.org>
References: <496A5CC0.8050605@american.edu> <glcc5i$qsh$1@ger.gmane.org>
Message-ID: <9457e7c80901230439h217be1f6g2905b584b2282982@mail.gmail.com>

Hi Tim

2009/1/22 Tim Michelsen <timmichelsen at gmx-topmail.de>:
> did you already come to a conclusion regarding this cite topic?
>
> Did you try to run the bibtext extension for Sphinx?

I haven't tried it.  One difficulty is that each docstring needs to be
self-contained, i.e., it must include its own references.  If you want
to maintain a central reference list, this list must either be
generated from the docstrings or, alternatively, the docstrings must
be built from the list.  I don't think the latter is an option really
(since it would require massive merges whenever a reference is
updated).

What it then comes down to, is that there is work to be done in the
documentation editor before we modify the standard.  Unfortunately, I
am flat broke when it comes to free time, but hopefully some other
people have more time to invest in the issue.

Regards,
St?fan


From faltet at pytables.org  Fri Jan 23 08:46:02 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 23 Jan 2009 14:46:02 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.1.1 available
Message-ID: <200901231446.02962.faltet@pytables.org>

==========================
 Announcing Numexpr 1.1.1
==========================

Numexpr is a fast numerical expression evaluator for NumPy.  With it,
expressions that operate on arrays (like "3*a+4*b") are accelerated
and use less memory than doing the same calculation in Python.

This is a bug-fix release, where the next issues have been addressed:

- The case for multidimensional array operands is properly accelerated
  now.  Added a new benchmark (based on a script provided by Andrew
  Collette, thanks!) for easily testing this case in the future.
  Closes #12.

- Added a fix to avoid the caches in numexpr to grow too much.  The
  dictionary caches are kept now always with less than 256 entries.
  Closes #11.

- The VERSION file is correctly copied now (support for it was not
  present in the 1.1 tar file, don't know exactly why).  Closes #8.

- The package is accessible now from PyPI, so that people from 'evil'
  countries can access it too.

In case you want to know more in detail what has changed in this
version, have a look at ``RELEASE_NOTES.txt`` in the tarball.


Where I can find Numexpr?
=========================

The project is hosted at Google code in:

http://code.google.com/p/numexpr/

And you can get the packages from PyPI as well:

http://pypi.python.org/pypi


How it works?
=============

See:

http://code.google.com/p/numexpr/wiki/Overview

for a detailed description by the original author of Numexpr (David M.
Cooke).


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.


Enjoy!

-- 
Francesc Alted


From dsdale24 at gmail.com  Fri Jan 23 11:53:00 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 23 Jan 2009 11:53:00 -0500
Subject: [Numpy-discussion] strange multiplication behavior with
	numpy.float64 and ndarray subclass
In-Reply-To: <a08e5f80901211007m6be7cc07qff9f8888d8182a87@mail.gmail.com>
References: <a08e5f80901210834u3d4fcfbahf969584ba71313a3@mail.gmail.com>
	<CCA1373B-4E2E-421D-94BC-2F5B907C4411@gmail.com>
	<a08e5f80901210905k3ca43f97sacfb492456ec2359@mail.gmail.com>
	<F3450F13-AC5D-4000-8812-919C303CE797@gmail.com>
	<a08e5f80901211007m6be7cc07qff9f8888d8182a87@mail.gmail.com>
Message-ID: <a08e5f80901230853m6a6e751ckead9fe687083f950@mail.gmail.com>

On Wed, Jan 21, 2009 at 1:07 PM, Darren Dale <dsdale24 at gmail.com> wrote:

>
>
> On Wed, Jan 21, 2009 at 12:26 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
>> > I dont understand why __array_priority__ is not being respected
>> > here. Ticket 826 lists the component as numpy.ma, it seems the
>> > problem is in numpy.core. I think the severity of the ticket should
>> > be increased. But I wasnt able to view the ticket, I keep getting an
>> > "internal server error".
>>
>> Ticket #826 bumped.
>>
>
> Just an additional bit of context. I'm working on a subclass that handles
> physical quantities, and issue 826 causes a quantity to be converted to a
> dimensionless magnitude.
>

I wonder if this issue is appearing in other places as well. Many of the
ndarray methods work without modification on my Quantity subclass, but the
methods that produce scalars do not. For instance, __getitem__ yields a
dimensionless number when called with an integer index, but  it yields
another Quantity if called with a range, so I have to reimplement
__getitem__ so it yields a quantity for single indices. tolist, min, max,
mean, are the same way. Is there an ndarray attribute I should be using to
tell the superclass what is the desired type (aside from
__array_priority__)?

Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090123/a80daa43/attachment.html>

From aisaac at american.edu  Fri Jan 23 11:58:36 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 23 Jan 2009 11:58:36 -0500
Subject: [Numpy-discussion] coding style: citations
In-Reply-To: <glcc5i$qsh$1@ger.gmane.org>
References: <496A5CC0.8050605@american.edu> <glcc5i$qsh$1@ger.gmane.org>
Message-ID: <4979F73C.40900@american.edu>

Tim Michelsen wrote:
> did you already come to a conclusion regarding this cite topic?
> Did you try to run the bibtext extension for Sphinx?
> If so, please update the documentation guidelines.


I hope we reached agreement that the documentation
should use reST citations and not reST footnotes.
You can use bib4txt to process *all* the modules Sphinx will
process and extract the citation references and construct
a references.  I explained why it is undesirable to use
`+` or `:` in the citation references.

I am not currently a Sphinx user.
What I can offer is to make useful changes
to bibstuff (including bib4txt) if these
are specfied.

Alan Isaac


From rmay31 at gmail.com  Fri Jan 23 16:31:23 2009
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 23 Jan 2009 15:31:23 -0600
Subject: [Numpy-discussion] Pattern for reading non-simple binary files
Message-ID: <497A372B.7000808@gmail.com>

Hi,

I'm trying to read in a data from a binary-formatted file. I have the data
format, (available at:
http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td7000.pdf if you're
really curious), but it's not what I would consider simple, with a lot of
different blocks and messages, some that are optional and some that have
different formats depending on the data type.  My question is, has anyone dealt
with data like this using numpy?  Have you found a good pattern for how to
construct a numpy dtype dynamically to decode the different parts of the file
appropriately as you go along?

Any insight would be appreciated.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From robert.kern at gmail.com  Fri Jan 23 17:07:54 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 23 Jan 2009 16:07:54 -0600
Subject: [Numpy-discussion] Pattern for reading non-simple binary files
In-Reply-To: <497A372B.7000808@gmail.com>
References: <497A372B.7000808@gmail.com>
Message-ID: <3d375d730901231407p61849805v9eff1bb8ab476b3e@mail.gmail.com>

On Fri, Jan 23, 2009 at 15:31, Ryan May <rmay31 at gmail.com> wrote:
> Hi,
>
> I'm trying to read in a data from a binary-formatted file. I have the data
> format, (available at:
> http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td7000.pdf if you're
> really curious), but it's not what I would consider simple, with a lot of
> different blocks and messages, some that are optional and some that have
> different formats depending on the data type.  My question is, has anyone dealt
> with data like this using numpy?

Yes!

> Have you found a good pattern for how to
> construct a numpy dtype dynamically to decode the different parts of the file
> appropriately as you go along?

I use mmap and create numpy arrays for each block using the ndarray
constructor with the appropriate offset parameter. There isn't much of
a pattern for constructing the dtypes except to use constructor
functions.

Good luck!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From efiring at hawaii.edu  Fri Jan 23 17:17:44 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 23 Jan 2009 12:17:44 -1000
Subject: [Numpy-discussion] Pattern for reading non-simple binary files
In-Reply-To: <497A372B.7000808@gmail.com>
References: <497A372B.7000808@gmail.com>
Message-ID: <497A4208.2060506@hawaii.edu>

Ryan May wrote:
> Hi,
> 
> I'm trying to read in a data from a binary-formatted file. I have the data
> format, (available at:
> http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td7000.pdf if you're
> really curious), but it's not what I would consider simple, with a lot of
> different blocks and messages, some that are optional and some that have
> different formats depending on the data type.  My question is, has anyone dealt
> with data like this using numpy?  Have you found a good pattern for how to
> construct a numpy dtype dynamically to decode the different parts of the file
> appropriately as you go along?
> 
> Any insight would be appreciated.
> 
> Ryan
> 

Ryan,

http://currents.soest.hawaii.edu/hg/hgwebdir.cgi/pycurrents/file/d7c5c9aac32d/adcp/rdiraw.py#l1

This gives an example of reading several related and rather complex 
binary file types generated by (oceanographic) acoustic Doppler current 
profilers.  I have not looked at the format you are dealing with, so I 
don't know if the methods I used are applicable to your case.

Eric


From millman at berkeley.edu  Fri Jan 23 20:29:31 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 23 Jan 2009 17:29:31 -0800
Subject: [Numpy-discussion] failure
In-Reply-To: <CB76AAD1-9AFA-4904-916B-FC8DA13CFB85@math.toronto.edu>
References: <CB76AAD1-9AFA-4904-916B-FC8DA13CFB85@math.toronto.edu>
Message-ID: <c7009a550901231729i15b89e91w53ce591f97fdad1@mail.gmail.com>

On Wed, Jan 21, 2009 at 10:53 AM, Gideon Simpson
<simpson at math.toronto.edu> wrote:
> ======================================================================
> FAIL: test_umath.TestComplexFunctions.test_against_cmath
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/nose/
> case.py", line 182, in runTest
>     self.test(*self.arg)
>   File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/
> numpy/core/tests/test_umath.py", line 268, in test_against_cmath
>     assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b)
> AssertionError: arcsinh -2j: (-1.31695789692-1.57079632679j); cmath:
> (1.31695789692-1.57079632679j)
>
> ----------------------------------------------------------------------
> Ran 1740 tests in 9.839s
>
> FAILED (KNOWNFAIL=1, failures=1)
> <nose.result.TextTestResult run=1740 errors=0 failures=1>
>
> How would you recommend I troubleshoot this?  How seriously should I
> take it?
>
> This is with a fresh Python 2.5.4 installation too.

I think it is a problem with cmath and should probably be marked as a
knownfailure:
http://bugs.python.org/issue1381


From simpson at math.toronto.edu  Sat Jan 24 11:21:31 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sat, 24 Jan 2009 11:21:31 -0500
Subject: [Numpy-discussion] numpy and the ACML
Message-ID: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>

Does anyone have a guide on how to get numpy to use the ACML as its  
blas/lapack backend?

-gideon



From njwilson23 at gmail.com  Sat Jan 24 13:34:08 2009
From: njwilson23 at gmail.com (Nat Wilson)
Date: Sat, 24 Jan 2009 13:34:08 -0500
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
Message-ID: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>

Would anyone be willing to help me interpret an error while trying to  
build and install Numpy? I've searched around, and haven't seen this  
elsewhere.

I've been running into this wall for about half the day now. I've  
tried reinstalling Python, using numpy 1.2.0 and 1.2.1.

I have Python 2.6.1, running on OS X 10.4.11, with a G4 PPC processor.

Here's the print out:

Ganymede:~/Desktop/numpy-1.2.1 username$ python setup.py build
Running from numpy source directory.
Traceback (most recent call last):
   File "setup.py", line 96, in <module>
     setup_package()
   File "setup.py", line 68, in setup_package
     from numpy.distutils.core import setup
   File "/Users/username/Desktop/numpy/numpy/distutils/__init__.py",  
line 6, in <module>
   File "/Users/username/Desktop/numpy/numpy/distutils/ccompiler.py",  
line 11, in <module>
   File "/Users/username/Desktop/numpy/numpy/distutils/log.py", line  
7, in <module>
   File "/Users/username/Desktop/numpy/numpy/distutils/misc_util.py",  
line 8, in <module>
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/tempfile.py", line 34, in <module>
     from random import Random as _Random
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/random.py", line 871, in <module>
     _inst = Random()
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/random.py", line 96, in __init__
     self.seed(x)
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/random.py", line 115, in seed
     super(Random, self).seed(a)
SystemError: error return without exception set

Any ideas?
I've had numpy/scipy installed in the past, but recently had to wipe  
everything and start from scratch. Everything should be pretty clean  
right now. Am I missing something obvious?

Thanks,
NJW


From nadavh at visionsense.com  Sat Jan 24 15:08:35 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 24 Jan 2009 22:08:35 +0200
Subject: [Numpy-discussion] numpy and the ACML
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
Message-ID: <710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>

You have setup.cfg:
* set add the directory where acml libraries reside to the library dir path list
* add acmllibraries under blas and lapack sections

  Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Gideon Simpson
????: ? 24-?????-09 18:21
??: Discussion of Numerical Python
????: [Numpy-discussion] numpy and the ACML
 
Does anyone have a guide on how to get numpy to use the ACML as its  
blas/lapack backend?

-gideon

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 2878 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090124/89bba0f9/attachment.bin>

From simpson at math.toronto.edu  Sat Jan 24 15:26:17 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sat, 24 Jan 2009 15:26:17 -0500
Subject: [Numpy-discussion] numpy and the ACML
In-Reply-To: <710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
Message-ID: <4242995C-7284-4654-B858-3D6C36460D71@math.toronto.edu>

Nadav-

That doesn't quite seem to work for me.  I added:

[blas_opt]
library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/include
libraries = acml

[lapack_opt]
library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/include
libraries = acml

to my site.cfg with no luck.  Somewhere else, people indicated that  
the ACML lacked a CBLAS which was necessary to make this work.
-gideon

On Jan 24, 2009, at 3:08 PM, Nadav Horesh wrote:

> You have setup.cfg:
> * set add the directory where acml libraries reside to the library  
> dir path list
> * add acmllibraries under blas and lapack sections
>
>  Nadav
>
>
> -----????? ??????-----
> ???: numpy-discussion-bounces at scipy.org ??? Gideon Simpson
> ????: ? 24-?????-09 18:21
> ??: Discussion of Numerical Python
> ????: [Numpy-discussion] numpy and the ACML
>
> Does anyone have a guide on how to get numpy to use the ACML as its
> blas/lapack backend?
>
> -gideon
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> <winmail.dat>_______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From ondrej at certik.cz  Sat Jan 24 16:00:58 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 24 Jan 2009 13:00:58 -0800
Subject: [Numpy-discussion] whohas command
Message-ID: <85b5c3130901241300v4b9bfdactc2962a9460ee297b@mail.gmail.com>

Hi,

I just discovered a new whohas command in Debian, that can show a
package in virtually all important distributions (Arch, Debian,
Fedora, Gentoo, openSUSE, Slackware (and
linuxpackages.net), Source  Mage,  Ubuntu, FreeBSD, NetBSD, OpenBSD,
Fink and MacPorts distributions).

So for example SymPy is in a lot of them already:

$ whohas sympy
Arch        python-sympy-git          20090121-1
    unsupported http://aur.archlinux.org/packages.php?ID=23341
MacPorts    py-sympy                  0.5.15

http://trac.macports.org/browser/trunk/dports/python/py-sympy/Portfile
MacPorts    py25-sympy                0.6.3

http://trac.macports.org/browser/trunk/dports/python/py25-sympy/Portfile
FreeBSD     py25-sympy                0.6.3
    math
http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-sympy
NetBSD      py-sympy                  0.5.15
    math        http://pkgsrc.se/math/py-sympy
Ubuntu      python-sympy              0.4.2-1
    universe    http://packages.ubuntu.com/gutsy/python-sympy
Ubuntu      python-sympy              0.5.8-1
    universe    http://packages.ubuntu.com/hardy/python-sympy
Ubuntu      python-sympy              0.5.15-1~hardy1
    universe
http://packages.ubuntu.com/hardy-backports/python-sympy
Ubuntu      python-sympy              0.5.15-1
    universe    http://packages.ubuntu.com/intrepid/python-sympy
Ubuntu      python-sympy              0.6.3-1
    universe    http://packages.ubuntu.com/jaunty/python-sympy
Debian      python-sympy              0.6.1-1
    testing     http://packages.debian.org/lenny/python-sympy
Fink        sympy-py24                0.5.6-1
                http://pdb.finkproject.org/pdb/package.php/sympy-py24
Fink        sympy-py25                0.5.6-1
                http://pdb.finkproject.org/pdb/package.php/sympy-py25
Gentoo      sympy                     0.6.3
                http://gentoo-portage.com/dev-python/sympy
Gentoo      sympy                     0.6.2
                http://gentoo-portage.com/dev-python/sympy
openSUSE    python-sympy              0.6.3
    science
http://packages.opensuse-community.org/packageinfo.jsp?checksum=a2258d3deaee5f411456f22ea1ab6bb7ec95e12c&amp;distro=openSUSE_111



It's also interesting to click on the links to get to the pages of
sympy for each distribution. And numpy:

$ whohas numpy
Arch        python-numexpr            1.1.1-1
    unsupported http://aur.archlinux.org/packages.php?ID=23380
Arch        python-numpy              1.2.1-3
    community   http://aur.archlinux.org/packages.php?ID=8497
Arch        scikits-audiolab-svn      1491-1
    unsupported http://aur.archlinux.org/packages.php?ID=21018
FreeBSD     py25-numpy                1.2.1,1
    math
http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-numpy
FreeBSD     py25-symeig               1.4_1
    math
http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-symeig
NetBSD      py-numpy                  1.1.0
    wip         http://pkgsrc.se/wip/py-numpy
NetBSD      py-numpy                  1.1.0
    math        http://pkgsrc.se/math/py-numpy
Ubuntu      python-numpy              1:1.0.1-8ubuntu1
    universe    http://packages.ubuntu.com/feisty/python-numpy
Ubuntu      python-numpy              1:1.0.3-1ubuntu2
    universe    http://packages.ubuntu.com/gutsy/python-numpy
Ubuntu      python-numpy              1:1.0.4-6ubuntu3
    universe    http://packages.ubuntu.com/hardy/python-numpy
Ubuntu      python-numpy              1:1.1.1-1
                http://packages.ubuntu.com/intrepid/python-numpy
Ubuntu      python-numpy              1:1.1.1-2ubuntu1
                http://packages.ubuntu.com/jaunty/python-numpy
Ubuntu      python-numpy-dbg          1:1.0.1-8ubuntu1
    universe    http://packages.ubuntu.com/feisty/python-numpy-dbg
Ubuntu      python-numpy-dbg          1:1.0.3-1ubuntu2
    universe    http://packages.ubuntu.com/gutsy/python-numpy-dbg
Ubuntu      python-numpy-dbg          1:1.0.4-6ubuntu3
    universe    http://packages.ubuntu.com/hardy/python-numpy-dbg
Ubuntu      python-numpy-dbg          1:1.1.1-1
                http://packages.ubuntu.com/intrepid/python-numpy-dbg
Ubuntu      python-numpy-dbg          1:1.1.1-2ubuntu1
                http://packages.ubuntu.com/jaunty/python-numpy-dbg
Ubuntu      python-numpy-dev          1:1.0.1-8ubuntu1
    universe    http://packages.ubuntu.com/feisty/python-numpy-dev
Ubuntu      python-numpy-dev          1:1.0.3-1ubuntu2
    universe    http://packages.ubuntu.com/gutsy/python-numpy-dev
Ubuntu      python-numpy-dev
                http://packages.ubuntu.com/hardy/python-numpy-dev
Ubuntu      python-numpy-doc          1:1.0.1-8ubuntu1
    universe    http://packages.ubuntu.com/feisty/python-numpy-doc
Ubuntu      python-numpy-doc          1:1.0.3-1ubuntu2
    universe    http://packages.ubuntu.com/gutsy/python-numpy-doc
Ubuntu      python-numpy-doc          1:1.0.4-6ubuntu3
    universe    http://packages.ubuntu.com/hardy/python-numpy-doc
Ubuntu      python-numpy-doc          1:1.1.1-1
                http://packages.ubuntu.com/intrepid/python-numpy-doc
Ubuntu      python-numpy-doc          1:1.1.1-2ubuntu1
                http://packages.ubuntu.com/jaunty/python-numpy-doc
Ubuntu      python-numpy-ext          1:1.0.1-8ubuntu1
    universe    http://packages.ubuntu.com/feisty/python-numpy-ext
MacPorts    py-numpy                  1.2.1

http://trac.macports.org/browser/trunk/dports/python/py-numpy/Portfile
MacPorts    py25-numpy                1.2.1

http://trac.macports.org/browser/trunk/dports/python/py25-numpy/Portfile
MacPorts    py26-numpy                1.2.1

http://trac.macports.org/browser/trunk/dports/python/py26-numpy/Portfile
Debian      python-numpy              1:1.0.1-1
    stable      http://packages.debian.org/etch/python-numpy
Debian      python-numpy-dev          1:1.0.1-1
    stable      http://packages.debian.org/etch/python-numpy-dev
Debian      python-numpy-doc          1:1.0.1-1
    stable      http://packages.debian.org/etch/python-numpy-doc
Debian      python-numpy-ext          1:1.0.1-1
    stable      http://packages.debian.org/etch/python-numpy-ext
Debian      python-numpy              1:1.1.0-3
    testing     http://packages.debian.org/lenny/python-numpy
Debian      python-numpy-dbg          1:1.1.0-3
    testing     http://packages.debian.org/lenny/python-numpy-dbg
Debian      python-numpy-dev
    testing     http://packages.debian.org/lenny/python-numpy-dev
Debian      python-numpy-doc          1:1.1.0-3
    testing     http://packages.debian.org/lenny/python-numpy-doc
Debian      python-numpy-ext          1:1.1.0-3
    testing     http://packages.debian.org/lenny/python-numpy-ext
Fink        dap-py25                  2.2.6.6-1
                http://pdb.finkproject.org/pdb/package.php/dap-py25
Fink        dap-py26                  2.2.6.6-1
                http://pdb.finkproject.org/pdb/package.php/dap-py26
Fink        pytables-py24             2.0.3-1

http://pdb.finkproject.org/pdb/package.php/pytables-py24
Fink        pytables-py25             2.0.3-1

http://pdb.finkproject.org/pdb/package.php/pytables-py25
Fink        scipy-core-py23           1.1.1-1

http://pdb.finkproject.org/pdb/package.php/scipy-core-py23
Fink        scipy-core-py24           1.1.1-1

http://pdb.finkproject.org/pdb/package.php/scipy-core-py24
Fink        scipy-core-py25           1.1.1-1

http://pdb.finkproject.org/pdb/package.php/scipy-core-py25
Fink        scipy-core-py26           1.1.1-1

http://pdb.finkproject.org/pdb/package.php/scipy-core-py26
Gentoo      numpy                     1.2.1
                http://gentoo-portage.com/dev-python/numpy
Gentoo      numpy                     1.1.1
                http://gentoo-portage.com/dev-python/numpy
Gentoo      numpy                     1.0.4-r2
                http://gentoo-portage.com/dev-python/numpy
openSUSE    python-numpy              1.2.1
    packman
http://packages.opensuse-community.org/packageinfo.jsp?checksum=f3f9a6df3da6c6996c98ec15bb42373c4eeb6b2d&amp;distro=openSUSE_111
openSUSE    python-numpy-debuginfo    1.2.1
    packman
http://packages.opensuse-community.org/packageinfo.jsp?checksum=fc6af8293ae8f29fd40263dcee69374255d348a7&amp;distro=openSUSE_111
openSUSE    python-numpy-debugsource  1.2.1
    packman
http://packages.opensuse-community.org/packageinfo.jsp?checksum=68c2cade8c8f2e85c1b3a67bad39982ef1c14e0b&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    games
http://packages.opensuse-community.org/packageinfo.jsp?checksum=dd9071d749be90039cf433498023f4370de62ea7&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.0
    hamradio
http://packages.opensuse-community.org/packageinfo.jsp?checksum=574963a98baede6f6628290431aeab75fcc6c967&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.0
    hamradio
http://packages.opensuse-community.org/packageinfo.jsp?checksum=528c925b6969de7b3d43556da107568a4473123c&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    ferichter/o
http://packages.opensuse-community.org/packageinfo.jsp?checksum=22490d0f9d03fb443f1c97d2b8ac7f49f950469d&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    ferichter/o
http://packages.opensuse-community.org/packageinfo.jsp?checksum=03018ce7d36dd90dc900b087f3f543fb0dc0192b&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    MadCAD/open
http://packages.opensuse-community.org/packageinfo.jsp?checksum=73f10047cc2a49b41e56dd7fea84d3e5c9efd735&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    MadCAD/open
http://packages.opensuse-community.org/packageinfo.jsp?checksum=0642cae514f7dda2745086907e4a9cdc0b7e5b49&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    werner2101/
http://packages.opensuse-community.org/packageinfo.jsp?checksum=99c6af1da69823faf16738df79d320e12ed99a30&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    werner2101/
http://packages.opensuse-community.org/packageinfo.jsp?checksum=0cd24f243e5d315ca0bd2a60032517888d12025f&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    science
http://packages.opensuse-community.org/packageinfo.jsp?checksum=1f1197793b043115249652d9c88d20466e3855ea&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    science
http://packages.opensuse-community.org/packageinfo.jsp?checksum=4ad91bcb9f9caa4a4aa668ffdd9d0c5949d823d7&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=a82655e40db6afe3525a497fd016a164f24a4488&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=0bd33a0c8c89e1ce52f4e4a1a2070a1e4ba7f631&amp;distro=openSUSE_111
openSUSE    python-numpy-debuginfo    1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=d079da08ab504d73bb69ca5a22095af546db00fe&amp;distro=openSUSE_111
openSUSE    python-numpy-debugsource  1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=ba9fd6e9352859a79ce51cc4b09119950a767aee&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    anubisg1/op
http://packages.opensuse-community.org/packageinfo.jsp?checksum=10d1efb12656b48e82065ea6228c3095e87f612c&amp;distro=openSUSE_111
openSUSE    python-numpy-debuginfo    1.2.1
    anubisg1/op
http://packages.opensuse-community.org/packageinfo.jsp?checksum=6439aa8a322bf6a28c80120fa4f653de33a012a0&amp;distro=openSUSE_111
openSUSE    python-numpy-debugsource  1.2.1
    anubisg1/op
http://packages.opensuse-community.org/packageinfo.jsp?checksum=ebc7ecee464db5487d28414597989e02124ef6f3&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=f47f40d1377ef66f2fe8a6bf6b9551235508ea5a&amp;distro=openSUSE_111
openSUSE    python-numpy-debuginfo    1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=b027d798f58c06e0814f65484562e3db3632108b&amp;distro=openSUSE_111
openSUSE    python-numpy-debugsource  1.2.1
    Education
http://packages.opensuse-community.org/packageinfo.jsp?checksum=d8c4d6aa5bcdf9aa3d946d87249c503a4219638a&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    thomas-schr
http://packages.opensuse-community.org/packageinfo.jsp?checksum=412bc59b5f64b2a7e17c47f493f3764789944311&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    thomas-schr
http://packages.opensuse-community.org/packageinfo.jsp?checksum=22b703695a8d00f2f4cea0981a02011ea89d7744&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.0
    cyberorg
http://packages.opensuse-community.org/packageinfo.jsp?checksum=569e248766d4671c7cb7e5ea204040b52449d317&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.0
    cyberorg
http://packages.opensuse-community.org/packageinfo.jsp?checksum=bc8e52b109de2be297e75562590e46e286fb8402&amp;distro=openSUSE_111
openSUSE    python-numpy-devel        1.2.1
    science
http://packages.opensuse-community.org/packageinfo.jsp?checksum=5d70fb2f3881a4e0621b9748d9b9e44a1c430d30&amp;distro=openSUSE_111
openSUSE    python-numpy              1.2.1
    science
http://packages.opensuse-community.org/packageinfo.jsp?checksum=84e24a6f7ff3228ac8a2e4a411548a15b38c6a1d&amp;distro=openSUSE_111
Source Mage numpy                     1.2.1
    test
Source Mage numpy                     1.2.1
    stable
Fedora      numpy                     1.2.0-1              2.2M
30-10-2008
OpenBSD     py-numpy                  1.0.4p0

http://www.openbsd.org/4.4_packages/i386/py-numpy-1.0.4p0.tgz-long.html


Which reminds me that I still haven't packaged the new numpy for
Debian (any help is appreciated).

Ondrej


From pav at iki.fi  Sat Jan 24 16:05:19 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 24 Jan 2009 21:05:19 +0000 (UTC)
Subject: [Numpy-discussion] numpy and the ACML
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
	<4242995C-7284-4654-B858-3D6C36460D71@math.toronto.edu>
Message-ID: <glfvqf$209$1@ger.gmane.org>

Sat, 24 Jan 2009 15:26:17 -0500, Gideon Simpson wrote:

> Nadav-
> 
> That doesn't quite seem to work for me.  I added:
> 
> [blas_opt]
> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/include
> libraries = acml
> 
> [lapack_opt]
> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/include
> libraries = acml
> 
> to my site.cfg with no luck.  Somewhere else, people indicated that the
> ACML lacked a CBLAS which was necessary to make this work.

Yep, IIRC you needed also CBLAS. (I think I got Numpy & Scipy linked 
against ACML at some point, but it's been a while and I've forgotten 
details...) There's a CBLAS here:

	http://www.netlib.org/blas/blast-forum/cblas.tgz

So, I think you need to compile it & link it with ACML, and add it in 
site.cfg with ACML libs.

-- 
Pauli Virtanen



From jdgleeson at mac.com  Sat Jan 24 16:38:29 2009
From: jdgleeson at mac.com (John Gleeson)
Date: Sat, 24 Jan 2009 14:38:29 -0700
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
Message-ID: <B9EE7B68-74AC-4A92-A80E-EFBA37AD5B33@mac.com>


On 2009-01-24, at 11:34 AM, Nat Wilson wrote:

> Would anyone be willing to help me interpret an error while trying to
> build and install Numpy? I've searched around, and haven't seen this
> elsewhere.
>
> I've been running into this wall for about half the day now. I've
> tried reinstalling Python, using numpy 1.2.0 and 1.2.1.
>
> I have Python 2.6.1, running on OS X 10.4.11, with a G4 PPC processor.
>

Have you tried setting the environment variable

MACOSX_DEPLOYMENT_TARGET=10.4

before building?


From njwilson23 at gmail.com  Sat Jan 24 16:58:53 2009
From: njwilson23 at gmail.com (Nat Wilson)
Date: Sat, 24 Jan 2009 16:58:53 -0500
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <B9EE7B68-74AC-4A92-A80E-EFBA37AD5B33@mac.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<B9EE7B68-74AC-4A92-A80E-EFBA37AD5B33@mac.com>
Message-ID: <6B994850-E077-49BA-BFEE-6F95A2A4C731@gmail.com>

I had not, but that doesn't seem to make any difference. By the way,  
I'm using gfortran 4.2.3 as I do this.

Nat

On Jan 24, 2009, at 4:38 PM, John Gleeson wrote:

>
> On 2009-01-24, at 11:34 AM, Nat Wilson wrote:
>
>> Would anyone be willing to help me interpret an error while trying to
>> build and install Numpy? I've searched around, and haven't seen this
>> elsewhere.
>>
>> I've been running into this wall for about half the day now. I've
>> tried reinstalling Python, using numpy 1.2.0 and 1.2.1.
>>
>> I have Python 2.6.1, running on OS X 10.4.11, with a G4 PPC  
>> processor.
>>
>
> Have you tried setting the environment variable
>
> MACOSX_DEPLOYMENT_TARGET=10.4
>
> before building?
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090124/d34ca3e2/attachment.html>

From jswhit at fastmail.fm  Sat Jan 24 17:24:34 2009
From: jswhit at fastmail.fm (Jeff Whitaker)
Date: Sat, 24 Jan 2009 15:24:34 -0700
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
Message-ID: <497B9522.4040005@fastmail.fm>

Nat Wilson wrote:
> Would anyone be willing to help me interpret an error while trying to  
> build and install Numpy? I've searched around, and haven't seen this  
> elsewhere.
>
> I've been running into this wall for about half the day now. I've  
> tried reinstalling Python, using numpy 1.2.0 and 1.2.1.
>
> I have Python 2.6.1, running on OS X 10.4.11, with a G4 PPC processor.
>
> Here's the print out:
>
> Ganymede:~/Desktop/numpy-1.2.1 username$ python setup.py build
> Running from numpy source directory.
> Traceback (most recent call last):
>    File "setup.py", line 96, in <module>
>      setup_package()
>    File "setup.py", line 68, in setup_package
>      from numpy.distutils.core import setup
>    File "/Users/username/Desktop/numpy/numpy/distutils/__init__.py",  
> line 6, in <module>
>    File "/Users/username/Desktop/numpy/numpy/distutils/ccompiler.py",  
> line 11, in <module>
>    File "/Users/username/Desktop/numpy/numpy/distutils/log.py", line  
> 7, in <module>
>    File "/Users/username/Desktop/numpy/numpy/distutils/misc_util.py",  
> line 8, in <module>
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/tempfile.py", line 34, in <module>
>      from random import Random as _Random
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/random.py", line 871, in <module>
>      _inst = Random()
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/random.py", line 96, in __init__
>      self.seed(x)
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/random.py", line 115, in seed
>      super(Random, self).seed(a)
> SystemError: error return without exception set
>
> Any ideas?
> I've had numpy/scipy installed in the past, but recently had to wipe  
> everything and start from scratch. Everything should be pretty clean  
> right now. Am I missing something obvious?
>
> Thanks,
> NJW
>
>   
numpy 1.2 doesn't work with python 2.6.  You'll either need to revert to 
python 2.5 or get the latest svn numpy (which still may have some python 
2.6 glitches).

-Jeff



From jswhit at fastmail.fm  Sat Jan 24 17:25:24 2009
From: jswhit at fastmail.fm (Jeff Whitaker)
Date: Sat, 24 Jan 2009 15:25:24 -0700
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
Message-ID: <497B9554.2080003@fastmail.fm>

Nat Wilson wrote:
> Would anyone be willing to help me interpret an error while trying to  
> build and install Numpy? I've searched around, and haven't seen this  
> elsewhere.
>
> I've been running into this wall for about half the day now. I've  
> tried reinstalling Python, using numpy 1.2.0 and 1.2.1.
>
> I have Python 2.6.1, running on OS X 10.4.11, with a G4 PPC processor.
>
> Here's the print out:
>
> Ganymede:~/Desktop/numpy-1.2.1 username$ python setup.py build
> Running from numpy source directory.
> Traceback (most recent call last):
>    File "setup.py", line 96, in <module>
>      setup_package()
>    File "setup.py", line 68, in setup_package
>      from numpy.distutils.core import setup
>    File "/Users/username/Desktop/numpy/numpy/distutils/__init__.py",  
> line 6, in <module>
>    File "/Users/username/Desktop/numpy/numpy/distutils/ccompiler.py",  
> line 11, in <module>
>    File "/Users/username/Desktop/numpy/numpy/distutils/log.py", line  
> 7, in <module>
>    File "/Users/username/Desktop/numpy/numpy/distutils/misc_util.py",  
> line 8, in <module>
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/tempfile.py", line 34, in <module>
>      from random import Random as _Random
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/random.py", line 871, in <module>
>      _inst = Random()
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/random.py", line 96, in __init__
>      self.seed(x)
>    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/random.py", line 115, in seed
>      super(Random, self).seed(a)
> SystemError: error return without exception set
>
> Any ideas?
> I've had numpy/scipy installed in the past, but recently had to wipe  
> everything and start from scratch. Everything should be pretty clean  
> right now. Am I missing something obvious?
>
> Thanks,
> NJW
>   

Nat:

numpy 1.2.x doesn't work with python 2.6.  You'll either need to revert 
to python 2.5 or get the latest svn numpy (which still may have some 
python 2.6 glitches).

-Jeff


From simpson at math.toronto.edu  Sat Jan 24 17:43:15 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sat, 24 Jan 2009 17:43:15 -0500
Subject: [Numpy-discussion] numpy and the ACML
In-Reply-To: <glfvqf$209$1@ger.gmane.org>
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
	<4242995C-7284-4654-B858-3D6C36460D71@math.toronto.edu>
	<glfvqf$209$1@ger.gmane.org>
Message-ID: <478644F8-4530-4ECE-BCB3-1900BE29FAE8@math.toronto.edu>

I've tried building CBLAS, which seems to run properly by itself, but  
numpy is still having difficulty.  I've got the following in my  
site.cfg:

[blas_opt]
library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/local/ 
nonsystem/simpson/acml4.2.0/gfortran64/lib
include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/ 
nonsystem/simpson/acml4.2.0/gfortran64/include
libraries = cblas, acml

[lapack_opt]
library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/local/ 
nonsystem/simpson/acml4.2.0/gfortran64/lib
include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/ 
nonsystem/simpson/acml4.2.0/gfortran64/include
libraries = cblas, acml

I also created a symbolic link in /usr/local/nonsystem/simpson/CBLAS/ 
lib/LINUX, from cblas_LINUX.a to libcblas.a.


Is there an easier way to check if numpy is locating the libs other  
than doing python setup.py build, and looking at the output?
-gideon

On Jan 24, 2009, at 4:05 PM, Pauli Virtanen wrote:

> Sat, 24 Jan 2009 15:26:17 -0500, Gideon Simpson wrote:
>
>> Nadav-
>>
>> That doesn't quite seem to work for me.  I added:
>>
>> [blas_opt]
>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/ 
>> include
>> libraries = acml
>>
>> [lapack_opt]
>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/ 
>> include
>> libraries = acml
>>
>> to my site.cfg with no luck.  Somewhere else, people indicated that  
>> the
>> ACML lacked a CBLAS which was necessary to make this work.
>
> Yep, IIRC you needed also CBLAS. (I think I got Numpy & Scipy linked
> against ACML at some point, but it's been a while and I've forgotten
> details...) There's a CBLAS here:
>
> 	http://www.netlib.org/blas/blast-forum/cblas.tgz
>
> So, I think you need to compile it & link it with ACML, and add it in
> site.cfg with ACML libs.
>
> -- 
> Pauli Virtanen
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From rmay31 at gmail.com  Sat Jan 24 17:58:12 2009
From: rmay31 at gmail.com (Ryan May)
Date: Sat, 24 Jan 2009 16:58:12 -0600
Subject: [Numpy-discussion] Bug with mafromtxt
Message-ID: <497B9D04.1050706@gmail.com>

Pierre,

I've found what I consider to be a bug in the new mafromtxt (though apparently it
existed in earlier versions as well).  If you have an entire column of data in a
file that contains only masked data, and try to get mafromtxt to automatically
choose the dtype, the dtype gets selected to be object type.  In this case, I'd
think the better behavior would be float, but I'm not sure how hard it would be
to make this the case.  Here's a test case:

import numpy as np
from StringIO import StringIO
s = StringIO('1 2 3\n4 5 6\n')
a = np.mafromtxt(s, missing='2,5', dtype=None)
print a.dtype

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From robert.kern at gmail.com  Sat Jan 24 18:09:41 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Jan 2009 17:09:41 -0600
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
Message-ID: <3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>

On Sat, Jan 24, 2009 at 12:34, Nat Wilson <njwilson23 at gmail.com> wrote:
> Would anyone be willing to help me interpret an error while trying to
> build and install Numpy? I've searched around, and haven't seen this
> elsewhere.

>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
> python2.6/tempfile.py", line 34, in <module>
>     from random import Random as _Random
>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
> python2.6/random.py", line 871, in <module>
>     _inst = Random()
>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
> python2.6/random.py", line 96, in __init__
>     self.seed(x)
>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
> python2.6/random.py", line 115, in seed
>     super(Random, self).seed(a)
> SystemError: error return without exception set
>
> Any ideas?

It looks unrelated to numpy. Can you import the random module in other
situations?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Sat Jan 24 18:10:22 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sat, 24 Jan 2009 18:10:22 -0500
Subject: [Numpy-discussion] Bug with mafromtxt
In-Reply-To: <497B9D04.1050706@gmail.com>
References: <497B9D04.1050706@gmail.com>
Message-ID: <1D1E7999-8D16-46C8-8A98-96B55FF8E5D5@gmail.com>

Ryan,
Thanks for reporting. An idea would be to force the dtype of the  
masked column to the largest dtype of the other columns (in your  
example, that would be int). I'll try to see how easily it can be done  
early next week. Meanwhile, you can always give an explicit dtype at  
creation.

On Jan 24, 2009, at 5:58 PM, Ryan May wrote:

> Pierre,
>
> I've found what I consider to be a bug in the new mafromtxt (though  
> apparently it
> existed in earlier versions as well).  If you have an entire column  
> of data in a
> file that contains only masked data, and try to get mafromtxt to  
> automatically
> choose the dtype, the dtype gets selected to be object type.  In  
> this case, I'd
> think the better behavior would be float, but I'm not sure how hard  
> it would be
> to make this the case.  Here's a test case:
>
> import numpy as np
> from StringIO import StringIO
> s = StringIO('1 2 3\n4 5 6\n')
> a = np.mafromtxt(s, missing='2,5', dtype=None)
> print a.dtype
>
> Ryan
>
> -- 
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From gnurser at googlemail.com  Sat Jan 24 18:18:31 2009
From: gnurser at googlemail.com (George Nurser)
Date: Sat, 24 Jan 2009 23:18:31 +0000
Subject: [Numpy-discussion] numpy and the ACML
In-Reply-To: <478644F8-4530-4ECE-BCB3-1900BE29FAE8@math.toronto.edu>
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
	<4242995C-7284-4654-B858-3D6C36460D71@math.toronto.edu>
	<glfvqf$209$1@ger.gmane.org>
	<478644F8-4530-4ECE-BCB3-1900BE29FAE8@math.toronto.edu>
Message-ID: <1d1e6ea70901241518p19abffcdt230cd24d58f74422@mail.gmail.com>

I did manage to get it working.
I remember that both libcblas.a (or a link to it) and libacml.so had
to be in the same directory.

Also I had to comment out lines 399-400 of setup.py:
#             if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]):
#                 return None # dotblas needs ATLAS, Fortran compiled
blas will not be sufficient

In my site.cfg I have
[blas]
blas_libs = cblas, acml
library_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/lib
include_dirs =  /noc/users/agn/ext/AMD64/acml/ifort64/include

[lapack]
language = f77
lapack_libs = acml
library_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/lib
include_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/include

Both libcblas.a (or a link to it) and libacml.so  are in
/noc/users/agn/ext/AMD64/acml/ifort64/lib

HTH. George.

2009/1/24 Gideon Simpson <simpson at math.toronto.edu>:
> I've tried building CBLAS, which seems to run properly by itself, but
> numpy is still having difficulty.  I've got the following in my
> site.cfg:
>
> [blas_opt]
> library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/local/
> nonsystem/simpson/acml4.2.0/gfortran64/lib
> include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/
> nonsystem/simpson/acml4.2.0/gfortran64/include
> libraries = cblas, acml
>
> [lapack_opt]
> library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/local/
> nonsystem/simpson/acml4.2.0/gfortran64/lib
> include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/
> nonsystem/simpson/acml4.2.0/gfortran64/include
> libraries = cblas, acml
>
> I also created a symbolic link in /usr/local/nonsystem/simpson/CBLAS/
> lib/LINUX, from cblas_LINUX.a to libcblas.a.
>
>
> Is there an easier way to check if numpy is locating the libs other
> than doing python setup.py build, and looking at the output?
> -gideon
>
> On Jan 24, 2009, at 4:05 PM, Pauli Virtanen wrote:
>
>> Sat, 24 Jan 2009 15:26:17 -0500, Gideon Simpson wrote:
>>
>>> Nadav-
>>>
>>> That doesn't quite seem to work for me.  I added:
>>>
>>> [blas_opt]
>>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
>>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>> include
>>> libraries = acml
>>>
>>> [lapack_opt]
>>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/lib
>>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>> include
>>> libraries = acml
>>>
>>> to my site.cfg with no luck.  Somewhere else, people indicated that
>>> the
>>> ACML lacked a CBLAS which was necessary to make this work.
>>
>> Yep, IIRC you needed also CBLAS. (I think I got Numpy & Scipy linked
>> against ACML at some point, but it's been a while and I've forgotten
>> details...) There's a CBLAS here:
>>
>>       http://www.netlib.org/blas/blast-forum/cblas.tgz
>>
>> So, I think you need to compile it & link it with ACML, and add it in
>> site.cfg with ACML libs.
>>
>> --
>> Pauli Virtanen
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From rmay31 at gmail.com  Sat Jan 24 18:23:02 2009
From: rmay31 at gmail.com (Ryan May)
Date: Sat, 24 Jan 2009 17:23:02 -0600
Subject: [Numpy-discussion] Bug with mafromtxt
In-Reply-To: <1D1E7999-8D16-46C8-8A98-96B55FF8E5D5@gmail.com>
References: <497B9D04.1050706@gmail.com>
	<1D1E7999-8D16-46C8-8A98-96B55FF8E5D5@gmail.com>
Message-ID: <497BA2D6.1040508@gmail.com>

Pierre GM wrote:
> Ryan,
> Thanks for reporting. An idea would be to force the dtype of the  
> masked column to the largest dtype of the other columns (in your  
> example, that would be int). I'll try to see how easily it can be done  
> early next week. Meanwhile, you can always give an explicit dtype at  
> creation.

Ok, thanks.  I've dug a little further, and it seems like the problem is that a
column of all missing values ends up as a column of all None's.  When you create
a (masked) array from a list of None's, you end up with an object array.  On one
hand I'd love for things to behave differently in this case, but on the other I
understand why things work this way.

Ryan

> 
> On Jan 24, 2009, at 5:58 PM, Ryan May wrote:
> 
>> Pierre,
>>
>> I've found what I consider to be a bug in the new mafromtxt (though  
>> apparently it
>> existed in earlier versions as well).  If you have an entire column  
>> of data in a
>> file that contains only masked data, and try to get mafromtxt to  
>> automatically
>> choose the dtype, the dtype gets selected to be object type.  In  
>> this case, I'd
>> think the better behavior would be float, but I'm not sure how hard  
>> it would be
>> to make this the case.  Here's a test case:
>>
>> import numpy as np
>> from StringIO import StringIO
>> s = StringIO('1 2 3\n4 5 6\n')
>> a = np.mafromtxt(s, missing='2,5', dtype=None)
>> print a.dtype
>>
>> Ryan
>>
>> -- 
>> Ryan May
>> Graduate Research Assistant
>> School of Meteorology
>> University of Oklahoma
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From simpson at math.toronto.edu  Sat Jan 24 18:43:50 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sat, 24 Jan 2009 18:43:50 -0500
Subject: [Numpy-discussion] numpy and the ACML
In-Reply-To: <1d1e6ea70901241518p19abffcdt230cd24d58f74422@mail.gmail.com>
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
	<4242995C-7284-4654-B858-3D6C36460D71@math.toronto.edu>
	<glfvqf$209$1@ger.gmane.org>
	<478644F8-4530-4ECE-BCB3-1900BE29FAE8@math.toronto.edu>
	<1d1e6ea70901241518p19abffcdt230cd24d58f74422@mail.gmail.com>
Message-ID: <659C9323-3FF1-49C5-B5DF-583084EB429D@math.toronto.edu>

That's not working for me.  Any thoughts on how to troubleshoot it?
-gideon

On Jan 24, 2009, at 6:18 PM, George Nurser wrote:

> I did manage to get it working.
> I remember that both libcblas.a (or a link to it) and libacml.so had
> to be in the same directory.
>
> Also I had to comment out lines 399-400 of setup.py:
> #             if ('NO_ATLAS_INFO',1) in  
> blas_info.get('define_macros',[]):
> #                 return None # dotblas needs ATLAS, Fortran compiled
> blas will not be sufficient
>
> In my site.cfg I have
> [blas]
> blas_libs = cblas, acml
> library_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/lib
> include_dirs =  /noc/users/agn/ext/AMD64/acml/ifort64/include
>
> [lapack]
> language = f77
> lapack_libs = acml
> library_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/lib
> include_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/include
>
> Both libcblas.a (or a link to it) and libacml.so  are in
> /noc/users/agn/ext/AMD64/acml/ifort64/lib
>
> HTH. George.
>
> 2009/1/24 Gideon Simpson <simpson at math.toronto.edu>:
>> I've tried building CBLAS, which seems to run properly by itself, but
>> numpy is still having difficulty.  I've got the following in my
>> site.cfg:
>>
>> [blas_opt]
>> library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/ 
>> local/
>> nonsystem/simpson/acml4.2.0/gfortran64/lib
>> include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/
>> nonsystem/simpson/acml4.2.0/gfortran64/include
>> libraries = cblas, acml
>>
>> [lapack_opt]
>> library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/ 
>> local/
>> nonsystem/simpson/acml4.2.0/gfortran64/lib
>> include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/
>> nonsystem/simpson/acml4.2.0/gfortran64/include
>> libraries = cblas, acml
>>
>> I also created a symbolic link in /usr/local/nonsystem/simpson/CBLAS/
>> lib/LINUX, from cblas_LINUX.a to libcblas.a.
>>
>>
>> Is there an easier way to check if numpy is locating the libs other
>> than doing python setup.py build, and looking at the output?
>> -gideon
>>
>> On Jan 24, 2009, at 4:05 PM, Pauli Virtanen wrote:
>>
>>> Sat, 24 Jan 2009 15:26:17 -0500, Gideon Simpson wrote:
>>>
>>>> Nadav-
>>>>
>>>> That doesn't quite seem to work for me.  I added:
>>>>
>>>> [blas_opt]
>>>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/ 
>>>> lib
>>>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>>> include
>>>> libraries = acml
>>>>
>>>> [lapack_opt]
>>>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/ 
>>>> lib
>>>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>>> include
>>>> libraries = acml
>>>>
>>>> to my site.cfg with no luck.  Somewhere else, people indicated that
>>>> the
>>>> ACML lacked a CBLAS which was necessary to make this work.
>>>
>>> Yep, IIRC you needed also CBLAS. (I think I got Numpy & Scipy linked
>>> against ACML at some point, but it's been a while and I've forgotten
>>> details...) There's a CBLAS here:
>>>
>>>      http://www.netlib.org/blas/blast-forum/cblas.tgz
>>>
>>> So, I think you need to compile it & link it with ACML, and add it  
>>> in
>>> site.cfg with ACML libs.
>>>
>>> --
>>> Pauli Virtanen
>>>
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From njwilson23 at gmail.com  Sat Jan 24 18:58:41 2009
From: njwilson23 at gmail.com (Nat Wilson)
Date: Sat, 24 Jan 2009 18:58:41 -0500
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
Message-ID: <A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>

Ah, no, I can't import it.

What does this mean? As far as I know, I built Python as instructed.


Python 2.6.1 (r261:67515, Jan 24 2009, 16:08:37)
[GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> from random import Random as _Random
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/random.py", line 871, in <module>
     _inst = Random()
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/random.py", line 96, in __init__
     self.seed(x)
   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/random.py", line 115, in seed
     super(Random, self).seed(a)
SystemError: error return without exception set

Nat


On Jan 24, 2009, at 6:09 PM, Robert Kern wrote:

> On Sat, Jan 24, 2009 at 12:34, Nat Wilson <njwilson23 at gmail.com>  
> wrote:
>> Would anyone be willing to help me interpret an error while trying to
>> build and install Numpy? I've searched around, and haven't seen this
>> elsewhere.
>
>>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
>> python2.6/tempfile.py", line 34, in <module>
>>     from random import Random as _Random
>>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
>> python2.6/random.py", line 871, in <module>
>>     _inst = Random()
>>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
>> python2.6/random.py", line 96, in __init__
>>     self.seed(x)
>>   File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
>> python2.6/random.py", line 115, in seed
>>     super(Random, self).seed(a)
>> SystemError: error return without exception set
>>
>> Any ideas?
>
> It looks unrelated to numpy. Can you import the random module in other
> situations?
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Sat Jan 24 19:12:20 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Jan 2009 18:12:20 -0600
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
Message-ID: <3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>

On Sat, Jan 24, 2009 at 17:58, Nat Wilson <njwilson23 at gmail.com> wrote:
> Ah, no, I can't import it.
>
> What does this mean? As far as I know, I built Python as instructed.

I'm not sure. Can you show me what os.urandom(16) gives you?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From njwilson23 at gmail.com  Sat Jan 24 19:31:36 2009
From: njwilson23 at gmail.com (Nat Wilson)
Date: Sat, 24 Jan 2009 19:31:36 -0500
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>
Message-ID: <0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>

It throws this out.

Python 2.6.1 (r261:67515, Jan 24 2009, 16:08:37)
[GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import os
 >>> os.urandom(16)
'\xe0;n\x8a*\xb4\x08N\x80<\xef\x9b*\x06\x1b\xc4'
 >>>

Nat

On Jan 24, 2009, at 7:12 PM, Robert Kern wrote:

> On Sat, Jan 24, 2009 at 17:58, Nat Wilson <njwilson23 at gmail.com>  
> wrote:
>> Ah, no, I can't import it.
>>
>> What does this mean? As far as I know, I built Python as instructed.
>
> I'm not sure. Can you show me what os.urandom(16) gives you?
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Sat Jan 24 19:41:49 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Jan 2009 18:41:49 -0600
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>
	<0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
Message-ID: <3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>

On Sat, Jan 24, 2009 at 18:31, Nat Wilson <njwilson23 at gmail.com> wrote:
> It throws this out.
>
> Python 2.6.1 (r261:67515, Jan 24 2009, 16:08:37)
> [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import os
>  >>> os.urandom(16)
> '\xe0;n\x8a*\xb4\x08N\x80<\xef\x9b*\x06\x1b\xc4'
>  >>>

Well, looking at the C code for random_seed(), I don't see a way for
it to return NULL without having an exception set (assuming that the
Python API calls aren't buggy). Except maybe the assert() call in
there. When you built your Python, are you sure that -DNDEBUG was
being used?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From njwilson23 at gmail.com  Sat Jan 24 20:13:25 2009
From: njwilson23 at gmail.com (Nat Wilson)
Date: Sat, 24 Jan 2009 20:13:25 -0500
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>
	<0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
	<3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
Message-ID: <99AB1E04-D4A2-4CBE-A0AE-08D0FE3D1CE7@gmail.com>

To be honest, I really don't know. I followed the directions in /Mac/ 
README for a framework install.

How would I check this?

Thanks a lot for sticking with me,

Nat

On Jan 24, 2009, at 7:41 PM, Robert Kern wrote:

> On Sat, Jan 24, 2009 at 18:31, Nat Wilson <njwilson23 at gmail.com>  
> wrote:
>> It throws this out.
>>
>> Python 2.6.1 (r261:67515, Jan 24 2009, 16:08:37)
>> [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
>> Type "help", "copyright", "credits" or "license" for more  
>> information.
>>>>> import os
>>>>> os.urandom(16)
>> '\xe0;n\x8a*\xb4\x08N\x80<\xef\x9b*\x06\x1b\xc4'
>>>>>
>
> Well, looking at the C code for random_seed(), I don't see a way for
> it to return NULL without having an exception set (assuming that the
> Python API calls aren't buggy). Except maybe the assert() call in
> there. When you built your Python, are you sure that -DNDEBUG was
> being used?
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Sat Jan 24 20:24:00 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Jan 2009 19:24:00 -0600
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <99AB1E04-D4A2-4CBE-A0AE-08D0FE3D1CE7@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>
	<0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
	<3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
	<99AB1E04-D4A2-4CBE-A0AE-08D0FE3D1CE7@gmail.com>
Message-ID: <3d375d730901241724t5bcc0895s852fadc572da02d5@mail.gmail.com>

On Sat, Jan 24, 2009 at 19:13, Nat Wilson <njwilson23 at gmail.com> wrote:
> To be honest, I really don't know. I followed the directions in /Mac/
> README for a framework install.
>
> How would I check this?

Rebuild and look at the compiler command lines as they go by.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Sat Jan 24 20:26:51 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Jan 2009 19:26:51 -0600
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <99AB1E04-D4A2-4CBE-A0AE-08D0FE3D1CE7@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>
	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>
	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>
	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>
	<0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
	<3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
	<99AB1E04-D4A2-4CBE-A0AE-08D0FE3D1CE7@gmail.com>
Message-ID: <3d375d730901241726o2e31d450h28b8728096bef18a@mail.gmail.com>

On Sat, Jan 24, 2009 at 19:13, Nat Wilson <njwilson23 at gmail.com> wrote:
> To be honest, I really don't know. I followed the directions in /Mac/
> README for a framework install.
>
> How would I check this?

Another possibility for tracking down the problem would be to start up
Python under gdb, add a breakpoint for the function random_seed() then
step through. That would help you find out the code path that makes it
return NULL without setting the exception.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From michael.abshoff at googlemail.com  Sat Jan 24 21:04:15 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 24 Jan 2009 18:04:15 -0800
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>	<0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
	<3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
Message-ID: <497BC89F.4090001@gmail.com>

Robert Kern wrote:
> On Sat, Jan 24, 2009 at 18:31, Nat Wilson <njwilson23 at gmail.com> wrote:
>> It throws this out.
>>
>> Python 2.6.1 (r261:67515, Jan 24 2009, 16:08:37)
>> [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>>  >>> import os
>>  >>> os.urandom(16)
>> '\xe0;n\x8a*\xb4\x08N\x80<\xef\x9b*\x06\x1b\xc4'
>>  >>>
> 
> Well, looking at the C code for random_seed(), I don't see a way for
> it to return NULL without having an exception set (assuming that the
> Python API calls aren't buggy). Except maybe the assert() call in
> there. When you built your Python, are you sure that -DNDEBUG was
> being used?
> 

Well, the gcc used to compiler Python is rather ancient and that gcc 
release by Apple has the reputation to be "buggier than a Florida swamp 
in July" and at least for building Sage it is blacklisted. So I would 
suggest updating gcc by using some more recent XCode and trying again.

Cheers,

Michael


From simpson at math.toronto.edu  Sat Jan 24 23:47:45 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sat, 24 Jan 2009 23:47:45 -0500
Subject: [Numpy-discussion] glibc error
Message-ID: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>

Having built an up to date lapack and ATLAS against gcc 4.3.2, I tried  
installing numpy 1.2.1 on Python 2.5.4.  When testing I get:

Python 2.5.4 (r254:67916, Jan 24 2009, 00:27:20)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy
 >>> numpy.test()
Running unit tests for numpy
NumPy version 1.2.1
NumPy is installed in /usr/local/nonsystem/simpson/lib/python2.5/site- 
packages/numpy
Python version 2.5.4 (r254:67916, Jan 24 2009, 00:27:20) [GCC 4.3.2]
nose version 0.10.4
..........................................................................................................................................................................................................................................................................................................................................................................................................................................F 
................K 
........................................................................ 
........................................................................ 
................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*** glibc 
  detected *** python: free(): invalid next size (fast):  
0x000000001196b550 ***


I then have to kill python to get control again.
-gideon



From david at ar.media.kyoto-u.ac.jp  Sat Jan 24 23:37:56 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 25 Jan 2009 13:37:56 +0900
Subject: [Numpy-discussion] glibc error
In-Reply-To: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
Message-ID: <497BECA4.8080301@ar.media.kyoto-u.ac.jp>

Gideon Simpson wrote:
> Having built an up to date lapack and ATLAS against gcc 4.3.2, I tried  
> installing numpy 1.2.1 on Python 2.5.4.  When testing I get:
>
> Python 2.5.4 (r254:67916, Jan 24 2009, 00:27:20)
> [GCC 4.3.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy
>  >>> numpy.test()
> Running unit tests for numpy
> NumPy version 1.2.1
> NumPy is installed in /usr/local/nonsystem/simpson/lib/python2.5/site- 
> packages/numpy
> Python version 2.5.4 (r254:67916, Jan 24 2009, 00:27:20) [GCC 4.3.2]
> nose version 0.10.4
> ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F 
> ................K 
> ........................................................................ 
> ........................................................................ 
> ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*** glibc 
>   detected *** python: free(): invalid next size (fast):  
> 0x000000001196b550 ***
>   

Can you rerun the test verbosely ?

python -c "import numpy; numpy.test(verbose=10)"

This glibc message is generally a symptom of serious memory corruption -
which is why only the OS can stop it at that point.

cheers,

David



From robert.kern at gmail.com  Sat Jan 24 23:54:22 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Jan 2009 22:54:22 -0600
Subject: [Numpy-discussion] glibc error
In-Reply-To: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
Message-ID: <3d375d730901242054h43f5abdbj308c4a6cee204589@mail.gmail.com>

On Sat, Jan 24, 2009 at 22:47, Gideon Simpson <simpson at math.toronto.edu> wrote:
> Having built an up to date lapack and ATLAS against gcc 4.3.2, I tried
> installing numpy 1.2.1 on Python 2.5.4.  When testing I get:

Run numpy.test(verbose=2) in order to have nose print out the name of
the method it is running before actually running it. This will let us
figure out which test is causing the failure.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From hoytak at cs.ubc.ca  Sat Jan 24 23:58:15 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Sat, 24 Jan 2009 20:58:15 -0800
Subject: [Numpy-discussion] glibc error
In-Reply-To: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
Message-ID: <4db580fd0901242058r6324d68btc917a27933ef5658@mail.gmail.com>

Not sure what the problem is in your case, but I had the same error a
while back.  For some reason, ATLAS thought it should be compiled as
64 bit but numpy as 32 bit. .  It went away after I passed the -b 32
flag to configure in ATLAS (I think).  Thus that's one thing to check.
Also, make sure you're using ATLAS 3.8.x, as I had problems with the
3.9 series (don't know if that would be an issue still).

--Hoyt

On Sat, Jan 24, 2009 at 8:47 PM, Gideon Simpson
<simpson at math.toronto.edu> wrote:
> Having built an up to date lapack and ATLAS against gcc 4.3.2, I tried
> installing numpy 1.2.1 on Python 2.5.4.  When testing I get:
>
> Python 2.5.4 (r254:67916, Jan 24 2009, 00:27:20)
> [GCC 4.3.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy
>  >>> numpy.test()
> Running unit tests for numpy
> NumPy version 1.2.1
> NumPy is installed in /usr/local/nonsystem/simpson/lib/python2.5/site-
> packages/numpy
> Python version 2.5.4 (r254:67916, Jan 24 2009, 00:27:20) [GCC 4.3.2]
> nose version 0.10.4
> ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F
> ................K
> ........................................................................
> ........................................................................
> ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*** glibc
>  detected *** python: free(): invalid next size (fast):
> 0x000000001196b550 ***
>
>
> I then have to kill python to get control again.
> -gideon
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++


From david at ar.media.kyoto-u.ac.jp  Sat Jan 24 23:48:32 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 25 Jan 2009 13:48:32 +0900
Subject: [Numpy-discussion] glibc error
In-Reply-To: <4db580fd0901242058r6324d68btc917a27933ef5658@mail.gmail.com>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
	<4db580fd0901242058r6324d68btc917a27933ef5658@mail.gmail.com>
Message-ID: <497BEF20.4000309@ar.media.kyoto-u.ac.jp>

Hoyt Koepke wrote:
> Not sure what the problem is in your case, but I had the same error a
> while back.  For some reason, ATLAS thought it should be compiled as
> 64 bit but numpy as 32 bit. .  It went away after I passed the -b 32
> flag to configure in ATLAS (I think).  Thus that's one thing to check.
> Also, make sure you're using ATLAS 3.8.x, as I had problems with the
> 3.9 series (don't know if that would be an issue still).
>   

Actually, I would advise using only 3.8.2. Previous versions had bugs
for some core routines used by numpy (at least 3.8.0 did). I am a bit
surprised that a 64 bits-built atlas would be runnable at all in a 32
bits binary - I would expect the link phase to fail if two different
object formats are linked together.

cheers,

David


From simpson at math.toronto.edu  Sun Jan 25 00:05:34 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sun, 25 Jan 2009 00:05:34 -0500
Subject: [Numpy-discussion] glibc error
In-Reply-To: <497BECA4.8080301@ar.media.kyoto-u.ac.jp>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
	<497BECA4.8080301@ar.media.kyoto-u.ac.jp>
Message-ID: <1BF61BD8-ED27-44A9-9A58-E7602869E964@math.toronto.edu>

Rerunning the tests with verbosity, it dies at:

test_single (test_linalg.TestSolve) ... ok
Ticket #652 ... *** glibc detected *** python: free(): invalid next  
size (fast): 0x0000000001e284e0 ***

I'm using ATLAS 3.8.2 and lapack 3.2.  ATLAS and lapack were all built  
with the -m64 flag.
-gideon

On Jan 24, 2009, at 11:37 PM, David Cournapeau wrote:

> Gideon Simpson wrote:
>> Having built an up to date lapack and ATLAS against gcc 4.3.2, I  
>> tried
>> installing numpy 1.2.1 on Python 2.5.4.  When testing I get:
>>
>> Python 2.5.4 (r254:67916, Jan 24 2009, 00:27:20)
>> [GCC 4.3.2] on linux2
>> Type "help", "copyright", "credits" or "license" for more  
>> information.
>>>>> import numpy
>>>>> numpy.test()
>> Running unit tests for numpy
>> NumPy version 1.2.1
>> NumPy is installed in /usr/local/nonsystem/simpson/lib/python2.5/ 
>> site-
>> packages/numpy
>> Python version 2.5.4 (r254:67916, Jan 24 2009, 00:27:20) [GCC 4.3.2]
>> nose version 0.10.4
>> ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F
>> ................K
>> ........................................................................
>> ........................................................................
>> ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*** glibc
>>  detected *** python: free(): invalid next size (fast):
>> 0x000000001196b550 ***
>>
>
> Can you rerun the test verbosely ?
>
> python -c "import numpy; numpy.test(verbose=10)"
>
> This glibc message is generally a symptom of serious memory  
> corruption -
> which is why only the OS can stop it at that point.
>
> cheers,
>
> David
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From david at ar.media.kyoto-u.ac.jp  Sat Jan 24 23:57:53 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 25 Jan 2009 13:57:53 +0900
Subject: [Numpy-discussion] glibc error
In-Reply-To: <1BF61BD8-ED27-44A9-9A58-E7602869E964@math.toronto.edu>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>	<497BECA4.8080301@ar.media.kyoto-u.ac.jp>
	<1BF61BD8-ED27-44A9-9A58-E7602869E964@math.toronto.edu>
Message-ID: <497BF151.7020209@ar.media.kyoto-u.ac.jp>

Gideon Simpson wrote:
> Rerunning the tests with verbosity, it dies at:
>
> test_single (test_linalg.TestSolve) ... ok
> Ticket #652 ... *** glibc detected *** python: free(): invalid next  
> size (fast): 0x0000000001e284e0 ***
>
> I'm using ATLAS 3.8.2 and lapack 3.2.
>   

Lapack 3.2 may be the problem. There have been interface changes in some
functions related to LU factorization in LAPACK 3.2 - I am not sure that
those are the ones used by eig, but since many people have run the test
suite with numpy 1.2.1 and atlas 3.8.2, I would look at that first.

cheers,

David


From matthew.brett at gmail.com  Sun Jan 25 04:35:25 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 25 Jan 2009 01:35:25 -0800
Subject: [Numpy-discussion] Suggested change for NaN,
	Infs float->int conversion
Message-ID: <1e2af89e0901250135h57aa07b6w52afde98a0060f71@mail.gmail.com>

Hi,

When converting arrays from float to ints, I notice that NaNs, Infs,
and -Infs all get the minimum integer value:

>>> flts = np.array([np.nan, np.inf, -np.inf])
>>> flts.astype(np.int16)
array([-32768, -32768, -32768], dtype=int16)

However, setting NaNs into integer arrays gives a value of 0

>>> ints = np.array([1])
>>> ints.dtype
dtype('int32')
>>> ints[0] = np.nan
>>> ints
array([0])

whereas Inf or -Inf raise an error (as Josef pointed out recently):

>> ints[0] = np.inf
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
OverflowError: cannot convert float infinity to long

>>> ints[0] = -np.inf
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
OverflowError: cannot convert float infinity to long

Matlab seems more consistent and sensible here:

>> flts = [NaN Inf -Inf];
>> int32(flts)

ans =

           0  2147483647 -2147483648

>> ints = int32([1 1 1]);
>> ints(:) = [NaN Inf -Inf]

ints =

           0  2147483647 -2147483648

Is there a route to change towards the matlab behavior?  Or at least
make numpy behavior self-consistent?

Best,

Matthew


From michael.abshoff at googlemail.com  Sun Jan 25 05:46:10 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sun, 25 Jan 2009 02:46:10 -0800
Subject: [Numpy-discussion] glibc error
In-Reply-To: <497BEF20.4000309@ar.media.kyoto-u.ac.jp>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>	<4db580fd0901242058r6324d68btc917a27933ef5658@mail.gmail.com>
	<497BEF20.4000309@ar.media.kyoto-u.ac.jp>
Message-ID: <497C42F2.60707@gmail.com>

David Cournapeau wrote:
> Hoyt Koepke wrote:

<SNIP>

> Actually, I would advise using only 3.8.2. Previous versions had bugs
> for some core routines used by numpy (at least 3.8.0 did). I am a bit
> surprised that a 64 bits-built atlas would be runnable at all in a 32
> bits binary - I would expect the link phase to fail if two different
> object formats are linked together.

Linking 32 and 64 bit ELF objects together in an extension will fail on 
any system but OSX where the ld will happily link together anything. 
Since that linker also does missing symbol lookup at runtime you will 
see some surprising distutils bugs when you thought that the build went 
perfectly, i.e. scipy 0.6 would not use the fortran compiler I would 
tell it to use, but one extension would use gfortran instead of 
sage_fortran when it was available in $PATH. sage_fortran would would 
just inject an "-m64" into the options and call gfortran. But with a few 
fortran objects being 32 bit some extensions in scipy would fail to 
import and it took me quite a while to track this one down. I haven't 
had time to test 0.7rc2 yet, but hopefully will do so in the next day or 
two.

> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From gnurser at googlemail.com  Sun Jan 25 06:34:38 2009
From: gnurser at googlemail.com (George Nurser)
Date: Sun, 25 Jan 2009 11:34:38 +0000
Subject: [Numpy-discussion] numpy and the ACML
In-Reply-To: <659C9323-3FF1-49C5-B5DF-583084EB429D@math.toronto.edu>
References: <B731964C-5072-4357-B71D-896DC6B81C25@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3DF@ex3.envision.co.il>
	<4242995C-7284-4654-B858-3D6C36460D71@math.toronto.edu>
	<glfvqf$209$1@ger.gmane.org>
	<478644F8-4530-4ECE-BCB3-1900BE29FAE8@math.toronto.edu>
	<1d1e6ea70901241518p19abffcdt230cd24d58f74422@mail.gmail.com>
	<659C9323-3FF1-49C5-B5DF-583084EB429D@math.toronto.edu>
Message-ID: <1d1e6ea70901250334me46f6bds1a12ede1e5405ae@mail.gmail.com>

Hmmm. I'm at a bit of a loss to understand why it isn't working for you.

How is it failing ?-- is the output of python setup.py build simply
giving blas not found, or is it something more complex?  Did you
remove the previous build directory, and the previous install
directory?

Have you changed the site.cfg in the numpy directory where setup.py
resides? -- this should be the important one -- and also in
numpy/distutils? --this shouldn't matter, but might be worth a try.

I'm no expert at debugging, but the class that's doing the work in
finding these libraries is deep in distutils:
 Configuration in numpy/distutils/misc_util.py

George.

2009/1/24 Gideon Simpson <simpson at math.toronto.edu>:
> That's not working for me.  Any thoughts on how to troubleshoot it?
> -gideon
>
> On Jan 24, 2009, at 6:18 PM, George Nurser wrote:
>
>> I did manage to get it working.
>> I remember that both libcblas.a (or a link to it) and libacml.so had
>> to be in the same directory.
>>
>> Also I had to comment out lines 399-400 of setup.py:
>> #             if ('NO_ATLAS_INFO',1) in
>> blas_info.get('define_macros',[]):
>> #                 return None # dotblas needs ATLAS, Fortran compiled
>> blas will not be sufficient
>>
>> In my site.cfg I have
>> [blas]
>> blas_libs = cblas, acml
>> library_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/lib
>> include_dirs =  /noc/users/agn/ext/AMD64/acml/ifort64/include
>>
>> [lapack]
>> language = f77
>> lapack_libs = acml
>> library_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/lib
>> include_dirs = /noc/users/agn/ext/AMD64/acml/ifort64/include
>>
>> Both libcblas.a (or a link to it) and libacml.so  are in
>> /noc/users/agn/ext/AMD64/acml/ifort64/lib
>>
>> HTH. George.
>>
>> 2009/1/24 Gideon Simpson <simpson at math.toronto.edu>:
>>> I've tried building CBLAS, which seems to run properly by itself, but
>>> numpy is still having difficulty.  I've got the following in my
>>> site.cfg:
>>>
>>> [blas_opt]
>>> library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/
>>> local/
>>> nonsystem/simpson/acml4.2.0/gfortran64/lib
>>> include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/
>>> nonsystem/simpson/acml4.2.0/gfortran64/include
>>> libraries = cblas, acml
>>>
>>> [lapack_opt]
>>> library_dirs = /usr/local/nonsystem/simpson/CBLAS/lib/LINUX:/usr/
>>> local/
>>> nonsystem/simpson/acml4.2.0/gfortran64/lib
>>> include_dirs = /usr/local/nonsystem/simpson/CBLAS/include:/usr/local/
>>> nonsystem/simpson/acml4.2.0/gfortran64/include
>>> libraries = cblas, acml
>>>
>>> I also created a symbolic link in /usr/local/nonsystem/simpson/CBLAS/
>>> lib/LINUX, from cblas_LINUX.a to libcblas.a.
>>>
>>>
>>> Is there an easier way to check if numpy is locating the libs other
>>> than doing python setup.py build, and looking at the output?
>>> -gideon
>>>
>>> On Jan 24, 2009, at 4:05 PM, Pauli Virtanen wrote:
>>>
>>>> Sat, 24 Jan 2009 15:26:17 -0500, Gideon Simpson wrote:
>>>>
>>>>> Nadav-
>>>>>
>>>>> That doesn't quite seem to work for me.  I added:
>>>>>
>>>>> [blas_opt]
>>>>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>>>> lib
>>>>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>>>> include
>>>>> libraries = acml
>>>>>
>>>>> [lapack_opt]
>>>>> library_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>>>> lib
>>>>> include_dirs = /usr/local/nonsystem/simpson/acml4.2.0/gfortran64/
>>>>> include
>>>>> libraries = acml
>>>>>
>>>>> to my site.cfg with no luck.  Somewhere else, people indicated that
>>>>> the
>>>>> ACML lacked a CBLAS which was necessary to make this work.
>>>>
>>>> Yep, IIRC you needed also CBLAS. (I think I got Numpy & Scipy linked
>>>> against ACML at some point, but it's been a while and I've forgotten
>>>> details...) There's a CBLAS here:
>>>>
>>>>      http://www.netlib.org/blas/blast-forum/cblas.tgz
>>>>
>>>> So, I think you need to compile it & link it with ACML, and add it
>>>> in
>>>> site.cfg with ACML libs.
>>>>
>>>> --
>>>> Pauli Virtanen
>>>>
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Numpy-discussion at scipy.org
>>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From stefan at sun.ac.za  Sun Jan 25 08:03:22 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 25 Jan 2009 15:03:22 +0200
Subject: [Numpy-discussion] Suggested change for NaN,
	Infs float->int 	conversion
In-Reply-To: <1e2af89e0901250135h57aa07b6w52afde98a0060f71@mail.gmail.com>
References: <1e2af89e0901250135h57aa07b6w52afde98a0060f71@mail.gmail.com>
Message-ID: <9457e7c80901250503g454763c8m9c7017d9bc85b688@mail.gmail.com>

2009/1/25 Matthew Brett <matthew.brett at gmail.com>:
> When converting arrays from float to ints, I notice that NaNs, Infs,
> and -Infs all get the minimum integer value:
>
>>>> flts = np.array([np.nan, np.inf, -np.inf])
>>>> flts.astype(np.int16)
> array([-32768, -32768, -32768], dtype=int16)
>
> However, setting NaNs into integer arrays gives a value of 0

I find this behaviour surprising too.  There is a ticket at

http://projects.scipy.org/scipy/numpy/ticket/980

Cheers
St?fan


From josef.pktd at gmail.com  Sun Jan 25 08:04:27 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 25 Jan 2009 08:04:27 -0500
Subject: [Numpy-discussion] Suggested change for NaN,
	Infs float->int 	conversion
In-Reply-To: <1e2af89e0901250135h57aa07b6w52afde98a0060f71@mail.gmail.com>
References: <1e2af89e0901250135h57aa07b6w52afde98a0060f71@mail.gmail.com>
Message-ID: <1cd32cbb0901250504p3f4f2e30tb7571f7f02320f09@mail.gmail.com>

On Sun, Jan 25, 2009 at 4:35 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> When converting arrays from float to ints, I notice that NaNs, Infs,
> and -Infs all get the minimum integer value:
>
>>>> flts = np.array([np.nan, np.inf, -np.inf])
>>>> flts.astype(np.int16)
> array([-32768, -32768, -32768], dtype=int16)
>
> However, setting NaNs into integer arrays gives a value of 0
>
>>>> ints = np.array([1])
>>>> ints.dtype
> dtype('int32')
>>>> ints[0] = np.nan
>>>> ints
> array([0])
>
> whereas Inf or -Inf raise an error (as Josef pointed out recently):
>
>>> ints[0] = np.inf
> ------------------------------------------------------------
> Traceback (most recent call last):
>  File "<ipython console>", line 1, in <module>
> OverflowError: cannot convert float infinity to long
>
>>>> ints[0] = -np.inf
> ------------------------------------------------------------
> Traceback (most recent call last):
>  File "<ipython console>", line 1, in <module>
> OverflowError: cannot convert float infinity to long
>
> Matlab seems more consistent and sensible here:
>
>>> flts = [NaN Inf -Inf];
>>> int32(flts)
>
> ans =
>
>           0  2147483647 -2147483648
>
>>> ints = int32([1 1 1]);
>>> ints(:) = [NaN Inf -Inf]
>
> ints =
>
>           0  2147483647 -2147483648
>
> Is there a route to change towards the matlab behavior?  Or at least
> make numpy behavior self-consistent?
>
> Best,
>
> Matthew


As we discussed in another thread, I think that the silent conversion
of nan to zero should not be done, since it is too much a source of
errors. Users should be forced to set nans to a valid number
explicitly.

Josef


From simpson at math.toronto.edu  Sun Jan 25 10:44:15 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sun, 25 Jan 2009 10:44:15 -0500
Subject: [Numpy-discussion] glibc error
In-Reply-To: <497C42F2.60707@gmail.com>
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>	<4db580fd0901242058r6324d68btc917a27933ef5658@mail.gmail.com>
	<497BEF20.4000309@ar.media.kyoto-u.ac.jp>
	<497C42F2.60707@gmail.com>
Message-ID: <E0EA6CEF-C270-497E-96C8-BA9E047CF55E@math.toronto.edu>

Rebuilding the library against ATLAS 3.8.2 with lapack 3.1.1 seems to  
have done the trick.  I do get one failure:

======================================================================
FAIL: test_umath.TestComplexFunctions.test_against_cmath
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/nose/ 
case.py", line 182, in runTest
     self.test(*self.arg)
   File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/ 
numpy/core/tests/test_umath.py", line 268, in test_against_cmath
     assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b)
AssertionError: arcsinh -2j: (-1.31695789692-1.57079632679j); cmath:  
(1.31695789692-1.57079632679j)

----------------------------------------------------------------------


-gideon

On Jan 25, 2009, at 5:46 AM, Michael Abshoff wrote:

> David Cournapeau wrote:
>> Hoyt Koepke wrote:
>
> <SNIP>
>
>> Actually, I would advise using only 3.8.2. Previous versions had bugs
>> for some core routines used by numpy (at least 3.8.0 did). I am a bit
>> surprised that a 64 bits-built atlas would be runnable at all in a 32
>> bits binary - I would expect the link phase to fail if two different
>> object formats are linked together.
>
> Linking 32 and 64 bit ELF objects together in an extension will fail  
> on
> any system but OSX where the ld will happily link together anything.
> Since that linker also does missing symbol lookup at runtime you will
> see some surprising distutils bugs when you thought that the build  
> went
> perfectly, i.e. scipy 0.6 would not use the fortran compiler I would
> tell it to use, but one extension would use gfortran instead of
> sage_fortran when it was available in $PATH. sage_fortran would would
> just inject an "-m64" into the options and call gfortran. But with a  
> few
> fortran objects being 32 bit some extensions in scipy would fail to
> import and it took me quite a while to track this one down. I haven't
> had time to test 0.7rc2 yet, but hopefully will do so in the next  
> day or
> two.
>
>> cheers,
>>
>> David
>
> Cheers,
>
> Michael
>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From njwilson23 at gmail.com  Sun Jan 25 12:13:31 2009
From: njwilson23 at gmail.com (Nat Wilson)
Date: Sun, 25 Jan 2009 12:13:31 -0500
Subject: [Numpy-discussion] Having trouble installing Numpy on OS X
In-Reply-To: <497BC89F.4090001@gmail.com>
References: <4BC33BAA-17F6-4BA4-9452-43FDC39DAED9@gmail.com>	<3d375d730901241509i69f0b8f2w802aae635a7446f1@mail.gmail.com>	<A39C5EF3-E605-4D87-B5F8-BE68DDE53CFF@gmail.com>	<3d375d730901241612o2c74b650hdb01535a86aa3269@mail.gmail.com>	<0D5AA8F9-AD2B-4681-A976-90F69733E149@gmail.com>
	<3d375d730901241641j31ef965ai6fcd43cd5bd0cbad@mail.gmail.com>
	<497BC89F.4090001@gmail.com>
Message-ID: <36A4F388-1C99-44CB-B8A1-5210F8EF7E1F@gmail.com>

I ended by giving up and grabbing the binary. I'll get 2.5 for Numpy  
too.

I did grab Xcode 2.5, which has gcc 4.0.1. I didn't feel up to  
manually doing a complete change to the latest gcc, at least while my  
needs are being met.

Thanks,

Nat

On Jan 24, 2009, at 9:04 PM, Michael Abshoff wrote:

> Robert Kern wrote:
>> On Sat, Jan 24, 2009 at 18:31, Nat Wilson <njwilson23 at gmail.com>  
>> wrote:
>>> It throws this out.
>>>
>>> Python 2.6.1 (r261:67515, Jan 24 2009, 16:08:37)
>>> [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more  
>>> information.
>>>>>> import os
>>>>>> os.urandom(16)
>>> '\xe0;n\x8a*\xb4\x08N\x80<\xef\x9b*\x06\x1b\xc4'
>>>>>>
>>
>> Well, looking at the C code for random_seed(), I don't see a way for
>> it to return NULL without having an exception set (assuming that the
>> Python API calls aren't buggy). Except maybe the assert() call in
>> there. When you built your Python, are you sure that -DNDEBUG was
>> being used?
>>
>
> Well, the gcc used to compiler Python is rather ancient and that gcc
> release by Apple has the reputation to be "buggier than a Florida  
> swamp
> in July" and at least for building Sage it is blacklisted. So I would
> suggest updating gcc by using some more recent XCode and trying again.
>
> Cheers,
>
> Michael
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pav at iki.fi  Sun Jan 25 12:14:01 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 25 Jan 2009 17:14:01 +0000 (UTC)
Subject: [Numpy-discussion] glibc error
References: <5A8A2038-9BAE-441B-9A5A-0760296E3F40@math.toronto.edu>
	<4db580fd0901242058r6324d68btc917a27933ef5658@mail.gmail.com>
	<497BEF20.4000309@ar.media.kyoto-u.ac.jp> <497C42F2.60707@gmail.com>
	<E0EA6CEF-C270-497E-96C8-BA9E047CF55E@math.toronto.edu>
Message-ID: <gli6kp$jj1$1@ger.gmane.org>

Sun, 25 Jan 2009 10:44:15 -0500, Gideon Simpson wrote:

> Rebuilding the library against ATLAS 3.8.2 with lapack 3.1.1 seems to
> have done the trick.  I do get one failure:
> 
> ======================================================================
> FAIL: test_umath.TestComplexFunctions.test_against_cmath
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/nose/
> case.py", line 182, in runTest
>      self.test(*self.arg)
>    File "/usr/local/nonsystem/simpson/lib/python2.5/site-packages/
> numpy/core/tests/test_umath.py", line 268, in test_against_cmath
>      assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b)
> AssertionError: arcsinh -2j: (-1.31695789692-1.57079632679j); cmath:
> (1.31695789692-1.57079632679j)

Same as http://scipy.org/scipy/numpy/ticket/977, fixed in trunk.

-- 
Pauli Virtanen



From pgmdevlist at gmail.com  Sun Jan 25 18:45:25 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 25 Jan 2009 18:45:25 -0500
Subject: [Numpy-discussion] Academic citation ?
Message-ID: <6553F040-65E9-4F4A-9469-1E0035FCA2CA@gmail.com>

All,
What is the most up-to-date way to cite Numpy and Scipy in an academic  
journal ?
Thanks a lot in advance
P.


From dwf at cs.toronto.edu  Sun Jan 25 20:17:47 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Sun, 25 Jan 2009 20:17:47 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <6553F040-65E9-4F4A-9469-1E0035FCA2CA@gmail.com>
References: <6553F040-65E9-4F4A-9469-1E0035FCA2CA@gmail.com>
Message-ID: <81C1C801-A65F-4BA5-A52F-71ECAC1A6EA6@cs.toronto.edu>

I believe this is what you're looking for:

	http://www.scipy.org/Citing_SciPy


On 25-Jan-09, at 6:45 PM, Pierre GM wrote:

> All,
> What is the most up-to-date way to cite Numpy and Scipy in an academic
> journal ?
> Thanks a lot in advance
> P.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pgmdevlist at gmail.com  Sun Jan 25 20:46:21 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 25 Jan 2009 20:46:21 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <81C1C801-A65F-4BA5-A52F-71ECAC1A6EA6@cs.toronto.edu>
References: <6553F040-65E9-4F4A-9469-1E0035FCA2CA@gmail.com>
	<81C1C801-A65F-4BA5-A52F-71ECAC1A6EA6@cs.toronto.edu>
Message-ID: <2DDDAA63-A025-401D-BC2F-67FFD5C30587@gmail.com>

David,
Thanks, but that's only part of what I need. I could also refer to  
Travis O's paper in Computing in Science and Engineering, but I  
wondered whether there wasn't something more up-to-date.
So, other answers are still welcome.
P.


On Jan 25, 2009, at 8:17 PM, David Warde-Farley wrote:

> I believe this is what you're looking for:
>
> 	http://www.scipy.org/Citing_SciPy
>
>
> On 25-Jan-09, at 6:45 PM, Pierre GM wrote:
>
>> All,
>> What is the most up-to-date way to cite Numpy and Scipy in an  
>> academic
>> journal ?
>> Thanks a lot in advance
>> P.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pgmdevlist at gmail.com  Sun Jan 25 20:50:06 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 25 Jan 2009 20:50:06 -0500
Subject: [Numpy-discussion] How to remove pages from the wiki ?
Message-ID: <54423B0F-E1DC-497F-ADCA-506318804B73@gmail.com>

All,
How can I remove pages from the wiki ? I ran into some that are  
clearly unrelated to numpy, scipy or programming...
Thanks a lot in advance,
P


From hanni.ali at gmail.com  Mon Jan 26 03:08:35 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 26 Jan 2009 08:08:35 +0000
Subject: [Numpy-discussion] List of Lists in C
Message-ID: <789d27b10901260008u56d74cf3x2ab34b0b4817bebf@mail.gmail.com>

Hi,

Quick question, I've been doing a fair bit of extension writing in C
recently, but wondered how best to implement:

>>> l = [[]] * 5

to create a list of a given length containing the initialization variable
desired.

A loop seems the straight forward manner, but I would have thought there was
a more efficient way...


Currently I just do it in Python and pass through the already initialized
list as it seems perfectly efficient.

Cheers,

Hanni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090126/f98b5edd/attachment.html>

From matthieu.brucher at gmail.com  Mon Jan 26 03:18:17 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 26 Jan 2009 09:18:17 +0100
Subject: [Numpy-discussion] List of Lists in C
In-Reply-To: <789d27b10901260008u56d74cf3x2ab34b0b4817bebf@mail.gmail.com>
References: <789d27b10901260008u56d74cf3x2ab34b0b4817bebf@mail.gmail.com>
Message-ID: <e76aa17f0901260018w32f9acah9678c3a1beb0a971@mail.gmail.com>

Hi,

Don't remember that you are using the same list in each element of the
outer list. If you don't want this, use [[] for i in range(5)]. I
don't think there is another way in C either (or too complicated).

Matthieu

2009/1/26 Hanni Ali <hanni.ali at gmail.com>:
> Hi,
>
> Quick question, I've been doing a fair bit of extension writing in C
> recently, but wondered how best to implement:
>
>>>> l = [[]] * 5
>
> to create a list of a given length containing the initialization variable
> desired.
>
> A loop seems the straight forward manner, but I would have thought there was
> a more efficient way...
>
>
> Currently I just do it in Python and pass through the already initialized
> list as it seems perfectly efficient.
>
> Cheers,
>
> Hanni
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From hanni.ali at gmail.com  Mon Jan 26 03:47:25 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 26 Jan 2009 08:47:25 +0000
Subject: [Numpy-discussion] List of Lists in C
In-Reply-To: <e76aa17f0901260018w32f9acah9678c3a1beb0a971@mail.gmail.com>
References: <789d27b10901260008u56d74cf3x2ab34b0b4817bebf@mail.gmail.com>
	<e76aa17f0901260018w32f9acah9678c3a1beb0a971@mail.gmail.com>
Message-ID: <789d27b10901260047v72b4a0fbp4607558c47b5c02f@mail.gmail.com>

Yes fair point, but when it's a empty list and new elements are replaced
with a new list instance it's fine, especially as [[]]*100000 is
significantly faster than [[] for i in xrange(100000)] as I was previously
doing.

In fact I think that's partly answered my question [[]]*x must create a list
of pointers pointing at the same list. Rather than [[] for i in
xrange(100000)] which must create a list of new separate lists instances.
Hence the significant difference in speed.

Hanni


2009/1/26 Matthieu Brucher <matthieu.brucher at gmail.com>

> Hi,
>
> Don't remember that you are using the same list in each element of the
> outer list. If you don't want this, use [[] for i in range(5)]. I
> don't think there is another way in C either (or too complicated).
>
> Matthieu
>
> 2009/1/26 Hanni Ali <hanni.ali at gmail.com>:
> > Hi,
> >
> > Quick question, I've been doing a fair bit of extension writing in C
> > recently, but wondered how best to implement:
> >
> >>>> l = [[]] * 5
> >
> > to create a list of a given length containing the initialization variable
> > desired.
> >
> > A loop seems the straight forward manner, but I would have thought there
> was
> > a more efficient way...
> >
> >
> > Currently I just do it in Python and pass through the already initialized
> > list as it seems perfectly efficient.
> >
> > Cheers,
> >
> > Hanni
> >
> >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
>
>
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090126/58f6f7f9/attachment.html>

From matthieu.brucher at gmail.com  Mon Jan 26 03:50:49 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 26 Jan 2009 09:50:49 +0100
Subject: [Numpy-discussion] List of Lists in C
In-Reply-To: <789d27b10901260047v72b4a0fbp4607558c47b5c02f@mail.gmail.com>
References: <789d27b10901260008u56d74cf3x2ab34b0b4817bebf@mail.gmail.com>
	<e76aa17f0901260018w32f9acah9678c3a1beb0a971@mail.gmail.com>
	<789d27b10901260047v72b4a0fbp4607558c47b5c02f@mail.gmail.com>
Message-ID: <e76aa17f0901260050w2d77ecb6w30d732ee4e418084@mail.gmail.com>

2009/1/26 Hanni Ali <hanni.ali at gmail.com>:
> Yes fair point, but when it's a empty list and new elements are replaced
> with a new list instance it's fine, especially as [[]]*100000 is
> significantly faster than [[] for i in xrange(100000)] as I was previously
> doing.

In this case, why do you put a list in it in the first place ? You
could put None, and it would be safer ;)

> In fact I think that's partly answered my question [[]]*x must create a list
> of pointers pointing at the same list. Rather than [[] for i in
> xrange(100000)] which must create a list of new separate lists instances.
> Hence the significant difference in speed.
>
> Hanni

I agree. Less memory allocations and initialization, thus more speed.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From hanni.ali at gmail.com  Mon Jan 26 04:01:57 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 26 Jan 2009 09:01:57 +0000
Subject: [Numpy-discussion] List of Lists in C
In-Reply-To: <e76aa17f0901260050w2d77ecb6w30d732ee4e418084@mail.gmail.com>
References: <789d27b10901260008u56d74cf3x2ab34b0b4817bebf@mail.gmail.com>
	<e76aa17f0901260018w32f9acah9678c3a1beb0a971@mail.gmail.com>
	<789d27b10901260047v72b4a0fbp4607558c47b5c02f@mail.gmail.com>
	<e76aa17f0901260050w2d77ecb6w30d732ee4e418084@mail.gmail.com>
Message-ID: <789d27b10901260101vc3b6091pace52f263835109c@mail.gmail.com>

Correct, however other areas of the application expect an empty list to be
present, otherwise I would have used None.

2009/1/26 Matthieu Brucher <matthieu.brucher at gmail.com>

> 2009/1/26 Hanni Ali <hanni.ali at gmail.com>:
> > Yes fair point, but when it's a empty list and new elements are replaced
> > with a new list instance it's fine, especially as [[]]*100000 is
> > significantly faster than [[] for i in xrange(100000)] as I was
> previously
> > doing.
>
> In this case, why do you put a list in it in the first place ? You
> could put None, and it would be safer ;)
>
> > In fact I think that's partly answered my question [[]]*x must create a
> list
> > of pointers pointing at the same list. Rather than [[] for i in
> > xrange(100000)] which must create a list of new separate lists instances.
> > Hence the significant difference in speed.
> >
> > Hanni
>
> I agree. Less memory allocations and initialization, thus more speed.
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090126/77840f29/attachment.html>

From boogaloojb at yahoo.fr  Mon Jan 26 09:59:35 2009
From: boogaloojb at yahoo.fr (Jean-Baptiste Rudant)
Date: Mon, 26 Jan 2009 14:59:35 +0000 (GMT)
Subject: [Numpy-discussion] Array of matrices - Inverse and dot
Message-ID: <932307.78047.qm@web28512.mail.ukl.yahoo.com>

Hello,

I would like to operate in an easy and efficient way (without python loop) with arrays of matrices.

Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like to calculate  inv_a and dot_ab, which would be arrays of N1*N2  (3*3)-matrices, such as :

inv_a[i, j] = np.linalg.inv(a[i, j])
dot_ab[i, j] = np.dot(a[i, j], b[i, j])

(where a and b could be  :
N1 = 5
N2 = 6
a = np.random((N1, N2, 3, 3)
b = np.random((N1, N2, 3, 3)
).

Thank you,

(Sorry to ask the list : I think it is quite a basic stuff, but searching for "array of matrices" on google didn't help me so much.)

Jean-Baptiste Rudant


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090126/6fe5d6e5/attachment.html>

From cournape at gmail.com  Mon Jan 26 10:18:33 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 27 Jan 2009 00:18:33 +0900
Subject: [Numpy-discussion] Array of matrices - Inverse and dot
In-Reply-To: <932307.78047.qm@web28512.mail.ukl.yahoo.com>
References: <932307.78047.qm@web28512.mail.ukl.yahoo.com>
Message-ID: <5b8d13220901260718h2bffb1aap4be4e4cf1feefbbf@mail.gmail.com>

On Mon, Jan 26, 2009 at 11:59 PM, Jean-Baptiste Rudant
<boogaloojb at yahoo.fr> wrote:
> Hello,
> I would like to operate in an easy and efficient way (without python
> loop) with arrays of matrices.
> Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like
> to calculate  inv_a and dot_ab, which would be arrays of N1*N2
>  (3*3)-matrices, such as :

If you only care about 3*3 matrices, I would consider using
"developed" formula of a 3x3 matrix inverse coded in numpy. numpy.inv
will be quite slow for such small matrices anyway (all the machinery
to call the underlying C code being almost certainly the bottleneck).

David


From boogaloojb at yahoo.fr  Mon Jan 26 10:25:54 2009
From: boogaloojb at yahoo.fr (Jean-Baptiste Rudant)
Date: Mon, 26 Jan 2009 15:25:54 +0000 (GMT)
Subject: [Numpy-discussion] Re :  Array of matrices - Inverse and dot
References: <932307.78047.qm@web28512.mail.ukl.yahoo.com>
	<5b8d13220901260718h2bffb1aap4be4e4cf1feefbbf@mail.gmail.com>
Message-ID: <344533.96607.qm@web28516.mail.ukl.yahoo.com>

Thank you, but my example was bad. I have to deal with matrices which can be 100*100. 



________________________________
De : David Cournapeau <cournape at gmail.com>
? : Discussion of Numerical Python <numpy-discussion at scipy.org>
Envoy? le : Lundi, 26 Janvier 2009, 16h18mn 33s
Objet : Re: [Numpy-discussion] Array of matrices - Inverse and dot

On Mon, Jan 26, 2009 at 11:59 PM, Jean-Baptiste Rudant
<boogaloojb at yahoo.fr> wrote:
> Hello,
> I would like to operate in an easy and efficient way (without python
> loop) with arrays of matrices.
> Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would like
> to calculate  inv_a and dot_ab, which would be arrays of N1*N2
>  (3*3)-matrices, such as :

If you only care about 3*3 matrices, I would consider using
"developed" formula of a 3x3 matrix inverse coded in numpy. numpy.inv
will be quite slow for such small matrices anyway (all the machinery
to call the underlying C code being almost certainly the bottleneck).

David
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090126/5ba9cb04/attachment.html>

From josef.pktd at gmail.com  Mon Jan 26 10:47:23 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 26 Jan 2009 10:47:23 -0500
Subject: [Numpy-discussion] Re : Array of matrices - Inverse and dot
In-Reply-To: <344533.96607.qm@web28516.mail.ukl.yahoo.com>
References: <932307.78047.qm@web28512.mail.ukl.yahoo.com>
	<5b8d13220901260718h2bffb1aap4be4e4cf1feefbbf@mail.gmail.com>
	<344533.96607.qm@web28516.mail.ukl.yahoo.com>
Message-ID: <1cd32cbb0901260747h31790850y7d00fca8558590e7@mail.gmail.com>

On Mon, Jan 26, 2009 at 10:25 AM, Jean-Baptiste Rudant
<boogaloojb at yahoo.fr> wrote:
> Thank you, but my example was bad. I have to deal with matrices which can be
> 100*100.
> ________________________________
> De : David Cournapeau <cournape at gmail.com>
> ? : Discussion of Numerical Python <numpy-discussion at scipy.org>
> Envoy? le : Lundi, 26 Janvier 2009, 16h18mn 33s
> Objet : Re: [Numpy-discussion] Array of matrices - Inverse and dot
>
> On Mon, Jan 26, 2009 at 11:59 PM, Jean-Baptiste Rudant
> <boogaloojb at yahoo.fr> wrote:
>> Hello,
>> I would like to operate in an easy and efficient way (without python
>> loop) with arrays of matrices.
>> Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would
>> like
>> to calculate  inv_a and dot_ab, which would be arrays of N1*N2
>>  (3*3)-matrices, such as :
>
> If you only care about 3*3 matrices, I would consider using
> "developed" formula of a 3x3 matrix inverse coded in numpy. numpy.inv
> will be quite slow for such small matrices anyway (all the machinery
> to call the underlying C code being almost certainly the bottleneck).
>
> David

There is numpy.tensordot and numpy.tensorinv which looks like it might
be doing what you want, but I never used it.

Josef


From jh at physics.ucf.edu  Mon Jan 26 10:56:27 2009
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Mon, 26 Jan 2009 10:56:27 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>
	(numpy-discussion-request@scipy.org)
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>
Message-ID: <wl8y6wyuklw.fsf@glup.physics.ucf.edu>

> What is the most up-to-date way to cite Numpy and Scipy in an academic  
> journal ?

Cite our conference articles here:

http://conference.scipy.org/proceedings/SciPy2008/index.html

It would be nice if someone involved in the proceedings could post a
bibtex on the citations page.  And link the citations page
to...something...easily navigated to from the front page.

This brings up a related point:

When someone goes to scipy.org, there is no way to navigate to
conferences.scipy.org from scipy.org except by finding the link buried
in the intro text.  Ipython and all the whatever.scipy.org domains,
except for docs.scipy.org, are completely absent; you have to know
about them to find them.  I don't even know where to find a complete
list of these.  They should all have a presence on at least the front
page and maybe the navigation.

--jh--


From gael.varoquaux at normalesup.org  Mon Jan 26 11:10:25 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 26 Jan 2009 17:10:25 +0100
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <wl8y6wyuklw.fsf@glup.physics.ucf.edu>
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>
	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>
Message-ID: <20090126161025.GH1894@phare.normalesup.org>

On Mon, Jan 26, 2009 at 10:56:27AM -0500, jh at physics.ucf.edu wrote:
> http://conference.scipy.org/proceedings/SciPy2008/index.html

> It would be nice if someone involved in the proceedings could post a
> bibtex on the citations page.

You want a bibtex entry that would be for all the proceedings? That may
be an idea. I wouldn't agree that this is similar to citing the numpy or
scipy authors, as Travis O and a few other did a huge amount of work, to
get these packages running, and they are not even in the proceedings. In
the long run, it would be great if the proceedings reflected well the
on-going contributions to the community. Right now things are still in
movement.

> And link the citations page to...something...easily navigated to from
> the front page.

That's easy.

> This brings up a related point:

> When someone goes to scipy.org, there is no way to navigate to
> conferences.scipy.org from scipy.org except by finding the link buried
> in the intro text.  Ipython and all the whatever.scipy.org domains,
> except for docs.scipy.org, are completely absent; you have to know
> about them to find them.  I don't even know where to find a complete
> list of these.  They should all have a presence on at least the front
> page and maybe the navigation.

Yes, that's a big issue. The front page needs to be completely reworked,
and probably not put as a wiki. Stefan had started some work on that, but
he has a thesis to write, right now.

Ga?l


From pgmdevlist at gmail.com  Mon Jan 26 11:27:29 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 26 Jan 2009 11:27:29 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <wl8y6wyuklw.fsf@glup.physics.ucf.edu>
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>
	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>
Message-ID: <915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>

JH,
Thx for the links, but I'm afraid I need something more basic than  
that. For example, I'm referring to Python as:

van Rossum, G. and Drake, F. L. (eds), 2006. Python Reference Manual,  
Python Software Foundation,. http://docs.python.org/ref/ref.html.

I could indeed use http://www.scipy.org/Citing_SciPy to cite Scipy  
(although the citation is incomplete), and define something similar  
for Numpy... Or refer to the "Computing in Science and Engineering"  
special issue. I'm just a bit surprised there's no official standard.

Thx,
P.



On Jan 26, 2009, at 10:56 AM, jh at physics.ucf.edu wrote:

>> What is the most up-to-date way to cite Numpy and Scipy in an  
>> academic
>> journal ?
>
> Cite our conference articles here:
>
> http://conference.scipy.org/proceedings/SciPy2008/index.html
>
> It would be nice if someone involved in the proceedings could post a
> bibtex on the citations page.  And link the citations page
> to...something...easily navigated to from the front page.
>
> This brings up a related point:
>
> When someone goes to scipy.org, there is no way to navigate to
> conferences.scipy.org from scipy.org except by finding the link buried
> in the intro text.  Ipython and all the whatever.scipy.org domains,
> except for docs.scipy.org, are completely absent; you have to know
> about them to find them.  I don't even know where to find a complete
> list of these.  They should all have a presence on at least the front
> page and maybe the navigation.
>
> --jh--
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From aisaac at american.edu  Mon Jan 26 12:43:29 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 26 Jan 2009 12:43:29 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>
	<915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>
Message-ID: <497DF641.10905@american.edu>

@MANUAL{ascher.dubois.hinsen.hugunin.oliphant-1999-np,
   author       = {Ascher, David and Paul F. Dubois and Konrad Hinsen and James Hugunin and Travis Oliphant},
   year         = 1999,
   title        = {Numerical Python},
   edition      = {UCRL-MA-128569},
   address      = {Livermore, CA},
   organization = {Lawrence Livermore National Laboratory},
   keywords = {numpy}
}

@ARTICLE{dubois.hinsen.hugunin-1996-cp,
   author  = {Dubois, Paul F. and Konrad Hinsen and James Hugunin},
   year    = {1996},
   title   = {Numerical Python},
   journal = {Computers in Physics},
   volume  = 10,
   number  = 3,
   month   = {May/June},
   keywords = {numpy}
}

@ARTICLE{dubois-1999-cse,
   author  = {Dubois, Paul F.},
   year    = 1999,
   title   = {Extending Python with Fortran},
   journal = {Computing Science and Engineering},
   volume  = 1,
   number  = 5,
   month   = {Sep/Oct},
   pages   = {66--73},
   keywords = {numpy}
}

@MANUAL{oliphant-2006-guide,
   author       = {Oliphant, Travis E.},
   year         = 2006,
   title        = {Guide to NumPy},
   month        = mar,
   address      = {Provo, UT},
   institution  = {Brigham Young University},
   url          = {http://www.tramy.us/},
   keywords     = {numpy}
}



From pgmdevlist at gmail.com  Mon Jan 26 12:48:53 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 26 Jan 2009 12:48:53 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <497DF641.10905@american.edu>
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>
	<915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>
	<497DF641.10905@american.edu>
Message-ID: <235F36F8-9CCA-4E31-9128-7C6CE099CBAB@gmail.com>

Alan,
Awesome, thx!
Now, what about the very last one ?

> @MANUAL{oliphant-2006-guide,
>   author       = {Oliphant, Travis E.},
>   year         = 2006,
>   title        = {Guide to NumPy},
>   month        = mar,
>   address      = {Provo, UT},
>   institution  = {Brigham Young University},
>   url          = {http://www.tramy.us/},
>   keywords     = {numpy}
> }

Shouldn't we refer to the new doc.scipy.org instead of Travis' site ?


From tpk at kraussfamily.org  Mon Jan 26 13:18:09 2009
From: tpk at kraussfamily.org (Tom K.)
Date: Mon, 26 Jan 2009 10:18:09 -0800 (PST)
Subject: [Numpy-discussion] Array of matrices - Inverse and dot
In-Reply-To: <932307.78047.qm@web28512.mail.ukl.yahoo.com>
References: <932307.78047.qm@web28512.mail.ukl.yahoo.com>
Message-ID: <21670624.post@talk.nabble.com>



Jean-Baptiste Rudant wrote:
> 
> I would like to operate in an easy and efficient way (without python loop)
> with arrays of matrices.
> 
> Suppose a and b are some arrays of N1*N2 matrices of size 3*3, I would
> like to calculate  inv_a and dot_ab, which would be arrays of N1*N2 
> (3*3)-matrices, such as :
> 
> inv_a[i, j] = np.linalg.inv(a[i, j])
> dot_ab[i, j] = np.dot(a[i, j], b[i, j])
> 
> (where a and b could be  :
> N1 = 5
> N2 = 6
> a = np.random((N1, N2, 3, 3)
> b = np.random((N1, N2, 3, 3)
> ).
> 
Here's a one-liner:
       numpy.array(map(numpy.dot, a, b))
that works for matrix multiply if a, b are (n, 3, 3).  Could do the same for
linalg.inv.

This comes up a lot in OFDM MIMO systems so I wrote C++ code for complex
matrix multiply (arbitrary size), 2x2 complex inverse and 2x2 complex matrix
singular values, and then swigged it. I know a colleague at work has
extended this work to arbitrary size inverse. I wish I could share the code
but it is something developed for my company which has fairly strict
policies about posting these things (not worth the red-tape)...

I vote "+1" for such features in Numpy.  I haven't looked too much "under
the hood" of numpy so I am not sure how you would do it or how hard it would
be.

Regards,
  Tom K.


-- 
View this message in context: http://www.nabble.com/Array-of-matrices---Inverse-and-dot-tp21666949p21670624.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From aisaac at american.edu  Mon Jan 26 14:10:46 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 26 Jan 2009 14:10:46 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <235F36F8-9CCA-4E31-9128-7C6CE099CBAB@gmail.com>
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>	<915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>	<497DF641.10905@american.edu>
	<235F36F8-9CCA-4E31-9128-7C6CE099CBAB@gmail.com>
Message-ID: <497E0AB6.9010303@american.edu>

On 1/26/2009 12:48 PM Pierre GM apparently wrote:
> Shouldn't we refer to the new doc.scipy.org instead of Travis' site ?


Not in my opinion: Travis wrote a book,
which is what is being cited.

The docs link is in fact to the same tramy site.

I must add that IMO, it would be a courtesy
for the docs to actually link to http://www.tramy.us/
rather than directly to the PDF.  I for one urge
my students to make the suggested donation when
they download the book, and I am disappointed that
the docs page renders this suggestion invisible.

Alan




From pav at iki.fi  Mon Jan 26 14:37:33 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 26 Jan 2009 19:37:33 +0000 (UTC)
Subject: [Numpy-discussion] Academic citation ?
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>
	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>
	<915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>
	<497DF641.10905@american.edu>
	<235F36F8-9CCA-4E31-9128-7C6CE099CBAB@gmail.com>
	<497E0AB6.9010303@american.edu>
Message-ID: <gll3dt$ths$1@ger.gmane.org>

Mon, 26 Jan 2009 14:10:46 -0500, Alan G Isaac wrote:

> On 1/26/2009 12:48 PM Pierre GM apparently wrote:
>> Shouldn't we refer to the new doc.scipy.org instead of Travis' site ?
> 
> 
> Not in my opinion: Travis wrote a book, which is what is being cited.
> 
> The docs link is in fact to the same tramy site.
> 
> I must add that IMO, it would be a courtesy for the docs to actually
> link to http://www.tramy.us/ rather than directly to the PDF.  I for one
> urge my students to make the suggested donation when they download the
> book, and I am disappointed that the docs page renders this suggestion
> invisible.

Fixed, no need to be dissappointed any more. It actually didn't cross my 
mind that there was a reason not to link directly to the PDF file, but I 
see now that there is.

-- 
Pauli Virtanen



From arokem at berkeley.edu  Mon Jan 26 14:48:58 2009
From: arokem at berkeley.edu (Ariel Rokem)
Date: Mon, 26 Jan 2009 11:48:58 -0800
Subject: [Numpy-discussion] Usage of numpy.where and pylab.find with strings
Message-ID: <FB2ADDCD-21C5-44FC-8E03-BC448700240E@berkeley.edu>

Hi - I am trying to find a string in a list with strings and have come  
across the following state of affairs:

In [228]: subjects
Out[228]:
['KAA',
  'CCS',
  'EJS',
  'MNM',
  'JHS',
  'LJL',
  'DVA',
  'FCL',
  'CNC',
  'KFM',
  'APM',
  'GMC']

In [229]: subjects[0]
Out[229]: 'KAA'

In [230]: subjects[0] == 'KAA'
Out[230]: True

In [231]: np.where(subjects == 'KAA')
Out[231]: ()

In [232]: pylab.find(subjects == 'KAA')
Out[232]: array([], dtype=int32)

It doesn't seem to matter if I make the list into an array:

In [233]: np.array(subjects)
Out[233]:
array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC',
        'KFM', 'APM', 'GMC'],
       dtype='|S3')

In [234]: pylab.find(subjects == 'KAA')
Out[234]: array([], dtype=int32)

In [235]: np.where(subjects == 'KAA')
Out[235]: ()

What am I doing wrong? What does it mean that the dtype is IS3?

Thanks a bunch --

Ariel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090126/f10433b9/attachment.html>

From robert.kern at gmail.com  Mon Jan 26 15:00:07 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 26 Jan 2009 14:00:07 -0600
Subject: [Numpy-discussion] Usage of numpy.where and pylab.find with
	strings
In-Reply-To: <FB2ADDCD-21C5-44FC-8E03-BC448700240E@berkeley.edu>
References: <FB2ADDCD-21C5-44FC-8E03-BC448700240E@berkeley.edu>
Message-ID: <3d375d730901261200s70c34a2fjfa33729f899d531f@mail.gmail.com>

On Mon, Jan 26, 2009 at 13:48, Ariel Rokem <arokem at berkeley.edu> wrote:
> Hi - I am trying to find a string in a list with strings and have come
> across the following state of affairs:
> In [228]: subjects
> Out[228]:
> ['KAA',
>  'CCS',
>  'EJS',
>  'MNM',
>  'JHS',
>  'LJL',
>  'DVA',
>  'FCL',
>  'CNC',
>  'KFM',
>  'APM',
>  'GMC']
> In [229]: subjects[0]
> Out[229]: 'KAA'
> In [230]: subjects[0] == 'KAA'
> Out[230]: True
> In [231]: np.where(subjects == 'KAA')
> Out[231]: ()
> In [232]: pylab.find(subjects == 'KAA')
> Out[232]: array([], dtype=int32)

Well, this will never work. Python lists don't broadcast like numpy arrays.

> It doesn't seem to matter if I make the list into an array:
> In [233]: np.array(subjects)
> Out[233]:
> array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC',
>        'KFM', 'APM', 'GMC'],
>       dtype='|S3')
> In [234]: pylab.find(subjects == 'KAA')
> Out[234]: array([], dtype=int32)
> In [235]: np.where(subjects == 'KAA')
> Out[235]: ()
> What am I doing wrong?

subjects is still a list. You did not assign the array to that name.

> What does it mean that the dtype is IS3?

All of your elements are length-3 strings.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From kwgoodman at gmail.com  Mon Jan 26 15:00:29 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 26 Jan 2009 12:00:29 -0800
Subject: [Numpy-discussion] Usage of numpy.where and pylab.find with
	strings
In-Reply-To: <FB2ADDCD-21C5-44FC-8E03-BC448700240E@berkeley.edu>
References: <FB2ADDCD-21C5-44FC-8E03-BC448700240E@berkeley.edu>
Message-ID: <f4f93d420901261200i6b732cc2g5369c42c7fc44ae8@mail.gmail.com>

On Mon, Jan 26, 2009 at 11:48 AM, Ariel Rokem <arokem at berkeley.edu> wrote:
> Hi - I am trying to find a string in a list with strings and have come
> across the following state of affairs:
> In [228]: subjects
> Out[228]:
> ['KAA',
>  'CCS',
>  'EJS',
>  'MNM',
>  'JHS',
>  'LJL',
>  'DVA',
>  'FCL',
>  'CNC',
>  'KFM',
>  'APM',
>  'GMC']
> In [229]: subjects[0]
> Out[229]: 'KAA'
> In [230]: subjects[0] == 'KAA'
> Out[230]: True
> In [231]: np.where(subjects == 'KAA')
> Out[231]: ()
> In [232]: pylab.find(subjects == 'KAA')
> Out[232]: array([], dtype=int32)
> It doesn't seem to matter if I make the list into an array:
> In [233]: np.array(subjects)
> Out[233]:
> array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC',
>        'KFM', 'APM', 'GMC'],
>       dtype='|S3')
> In [234]: pylab.find(subjects == 'KAA')
> Out[234]: array([], dtype=int32)
> In [235]: np.where(subjects == 'KAA')
> Out[235]: ()
> What am I doing wrong? What does it mean that the dtype is IS3?

I think you made a typo. Try changing line 233 to subjects = np.array(subjects)

>> type(subjects)
   <type 'list'>
>> np.where(subjects == 'KAA')
   ()
>> np.where(np.asarray(subjects) == 'KAA')
   (array([0]),)


From aisaac at american.edu  Mon Jan 26 15:02:07 2009
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 26 Jan 2009 15:02:07 -0500
Subject: [Numpy-discussion] Academic citation ?
In-Reply-To: <gll3dt$ths$1@ger.gmane.org>
References: <mailman.4784.1232960524.2878.numpy-discussion@scipy.org>	<wl8y6wyuklw.fsf@glup.physics.ucf.edu>	<915BBEE3-0FC8-48D5-937E-F957F77D373E@gmail.com>	<497DF641.10905@american.edu>	<235F36F8-9CCA-4E31-9128-7C6CE099CBAB@gmail.com>	<497E0AB6.9010303@american.edu>
	<gll3dt$ths$1@ger.gmane.org>
Message-ID: <497E16BF.4080807@american.edu>

On 1/26/2009 2:37 PM Pauli Virtanen apparently wrote:
 > Fixed, no need to be dissappointed any more.

Thanks!
Alan
PS I hope "disappointed" did not sound so strong
as to be discourteous.  Email is tricky, and I
tend to write mine with dangerous speed.



From pgmdevlist at gmail.com  Mon Jan 26 16:07:00 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 26 Jan 2009 16:07:00 -0500
Subject: [Numpy-discussion] Bug with mafromtxt
In-Reply-To: <497BA2D6.1040508@gmail.com>
References: <497B9D04.1050706@gmail.com>
	<1D1E7999-8D16-46C8-8A98-96B55FF8E5D5@gmail.com>
	<497BA2D6.1040508@gmail.com>
Message-ID: <96627BB8-85FD-424D-81AE-A7B7288EC9AF@gmail.com>


On Jan 24, 2009, at 6:23 PM, Ryan May wrote:
>>
>
> Ok, thanks.  I've dug a little further, and it seems like the  
> problem is that a
> column of all missing values ends up as a column of all None's.   
> When you create
> a (masked) array from a list of None's, you end up with an object  
> array.  On one
> hand I'd love for things to behave differently in this case, but on  
> the other I
> understand why things work this way.

Ryan,
Mind giving r6434 a try? As usual, don't hesitate to report any problem.


From rmay31 at gmail.com  Mon Jan 26 16:26:43 2009
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 26 Jan 2009 15:26:43 -0600
Subject: [Numpy-discussion] Bug with mafromtxt
In-Reply-To: <96627BB8-85FD-424D-81AE-A7B7288EC9AF@gmail.com>
References: <497B9D04.1050706@gmail.com>	<1D1E7999-8D16-46C8-8A98-96B55FF8E5D5@gmail.com>	<497BA2D6.1040508@gmail.com>
	<96627BB8-85FD-424D-81AE-A7B7288EC9AF@gmail.com>
Message-ID: <497E2A93.8070102@gmail.com>

Pierre GM wrote:
> On Jan 24, 2009, at 6:23 PM, Ryan May wrote:
>> Ok, thanks.  I've dug a little further, and it seems like the  
>> problem is that a
>> column of all missing values ends up as a column of all None's.   
>> When you create
>> a (masked) array from a list of None's, you end up with an object  
>> array.  On one
>> hand I'd love for things to behave differently in this case, but on  
>> the other I
>> understand why things work this way.
> 
> Ryan,
> Mind giving r6434 a try? As usual, don't hesitate to report any problem.

Works great! Thanks for the quick fix.  I had racked my brain on how to go about
fixing this cleanly, but this is far simpler than what I would have done.  It
makes sense, since all I really needed for the masked column was something
*other* than object.

Thanks a lot,

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From arokem at berkeley.edu  Mon Jan 26 18:18:42 2009
From: arokem at berkeley.edu (Ariel Rokem)
Date: Mon, 26 Jan 2009 15:18:42 -0800
Subject: [Numpy-discussion] Usage of numpy.where and pylab.find with
	strings
In-Reply-To: <f4f93d420901261200i6b732cc2g5369c42c7fc44ae8@mail.gmail.com>
References: <FB2ADDCD-21C5-44FC-8E03-BC448700240E@berkeley.edu>
	<f4f93d420901261200i6b732cc2g5369c42c7fc44ae8@mail.gmail.com>
Message-ID: <CD25B31C-4F4E-4E0D-8258-FA98B5721A82@berkeley.edu>

Doh! That's embarrassing!
Thanks!

On Jan 26, 2009, at 12:00 PM, Keith Goodman wrote:

> On Mon, Jan 26, 2009 at 11:48 AM, Ariel Rokem <arokem at berkeley.edu>  
> wrote:
>> Hi - I am trying to find a string in a list with strings and have  
>> come
>> across the following state of affairs:
>> In [228]: subjects
>> Out[228]:
>> ['KAA',
>> 'CCS',
>> 'EJS',
>> 'MNM',
>> 'JHS',
>> 'LJL',
>> 'DVA',
>> 'FCL',
>> 'CNC',
>> 'KFM',
>> 'APM',
>> 'GMC']
>> In [229]: subjects[0]
>> Out[229]: 'KAA'
>> In [230]: subjects[0] == 'KAA'
>> Out[230]: True
>> In [231]: np.where(subjects == 'KAA')
>> Out[231]: ()
>> In [232]: pylab.find(subjects == 'KAA')
>> Out[232]: array([], dtype=int32)
>> It doesn't seem to matter if I make the list into an array:
>> In [233]: np.array(subjects)
>> Out[233]:
>> array(['KAA', 'CCS', 'EJS', 'MNM', 'JHS', 'LJL', 'DVA', 'FCL', 'CNC',
>>       'KFM', 'APM', 'GMC'],
>>      dtype='|S3')
>> In [234]: pylab.find(subjects == 'KAA')
>> Out[234]: array([], dtype=int32)
>> In [235]: np.where(subjects == 'KAA')
>> Out[235]: ()
>> What am I doing wrong? What does it mean that the dtype is IS3?
>
> I think you made a typo. Try changing line 233 to subjects =  
> np.array(subjects)
>
>>> type(subjects)
>   <type 'list'>
>>> np.where(subjects == 'KAA')
>   ()
>>> np.where(np.asarray(subjects) == 'KAA')
>   (array([0]),)
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From cycomanic at gmail.com  Mon Jan 26 19:26:21 2009
From: cycomanic at gmail.com (Jochen)
Date: Tue, 27 Jan 2009 13:26:21 +1300
Subject: [Numpy-discussion] numpy ndarray questions
Message-ID: <1233015981.4180.39.camel@phy.auckland.ac.nz>

Hi all,

I just wrote ctypes bindings to fftw3 (see
http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
for the post to scipy). 
Now I have a couple of numpy related questions:

In order to be able to use simd instructions I 
create an ndarray subclass, which uses fftw_malloc to allocate the
memory and fftw_free to free the memory when the array is deleted. This
works fine for inplace operations however if someone does something like
this:

a = fftw3.AlignedArray(1024,complex)

a = a+1

a.ctypes.data points to a different memory location (this is actually an
even bigger problem when executing fftw plans), however 
type(a) still gives me <class 'fftw3.planning.AlignedArray'>.

I think I understand the reason for this is that python does a copy
internally (it does not call the __copy__ or __deepcopy__ methods?). Is
there a way that I get a different object type?  Or even better is there
a way to prevent operations like a=a+1 or make them automatically
in-place operations? I realise that I could change the __add__ ...
methods but that would also prevent b=a+1 operations. 

My second comment is with respect to the documentation for
numpy.ctypeslib.ndpointer 
The documentation says:
An ndpointer instance is used to describe an ndarray in restypes
and argtypes specifications.

however if I specify a restype to a function e.g.
clib.malloc.restype =
ctypeslib.ndpointer(flags='contiguous,aligned',shape=shape,dtype=dtype)
Calling clib.malloc(1024) results in a TypeError:
TypeError: default __new__ takes no parameters

Am I correct in assuming that the documentation is incorrect and
ndpointer can only be used for argtypes? Or am I missing something?


Cheers
Jochen



From rmay31 at gmail.com  Mon Jan 26 20:25:44 2009
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 26 Jan 2009 19:25:44 -0600
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233015981.4180.39.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
Message-ID: <497E6298.7070708@gmail.com>

Jochen wrote:
> Hi all,
> 
> I just wrote ctypes bindings to fftw3 (see
> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> for the post to scipy). 
> Now I have a couple of numpy related questions:
> 
> In order to be able to use simd instructions I 
> create an ndarray subclass, which uses fftw_malloc to allocate the
> memory and fftw_free to free the memory when the array is deleted. This
> works fine for inplace operations however if someone does something like
> this:
> 
> a = fftw3.AlignedArray(1024,complex)
> 
> a = a+1
> 
> a.ctypes.data points to a different memory location (this is actually an
> even bigger problem when executing fftw plans), however 
> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.

This might help some:

http://www.scipy.org/Subclasses

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From cycomanic at gmail.com  Mon Jan 26 20:46:16 2009
From: cycomanic at gmail.com (Jochen)
Date: Tue, 27 Jan 2009 14:46:16 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497E6298.7070708@gmail.com>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497E6298.7070708@gmail.com>
Message-ID: <1233020776.20296.4.camel@phy.auckland.ac.nz>

On Mon, 2009-01-26 at 19:25 -0600, Ryan May wrote:
> Jochen wrote:
> > Hi all,
> > 
> > I just wrote ctypes bindings to fftw3 (see
> > http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> > for the post to scipy). 
> > Now I have a couple of numpy related questions:
> > 
> > In order to be able to use simd instructions I 
> > create an ndarray subclass, which uses fftw_malloc to allocate the
> > memory and fftw_free to free the memory when the array is deleted. This
> > works fine for inplace operations however if someone does something like
> > this:
> > 
> > a = fftw3.AlignedArray(1024,complex)
> > 
> > a = a+1
> > 
> > a.ctypes.data points to a different memory location (this is actually an
> > even bigger problem when executing fftw plans), however 
> > type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
> 
> This might help some:
> 
> http://www.scipy.org/Subclasses
> 
> Ryan
> 
Thanks, I had read about __array_finalize__, but not about
__array_wrap__. I'm not quite sure if I understand how I can use this to
get around my problem, can you explain?

Cheers
Jochen




From david at ar.media.kyoto-u.ac.jp  Mon Jan 26 22:49:17 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Jan 2009 12:49:17 +0900
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233015981.4180.39.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
Message-ID: <497E843D.6080005@ar.media.kyoto-u.ac.jp>

Jochen wrote:
> Hi all,
>
> I just wrote ctypes bindings to fftw3 (see
> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> for the post to scipy). 
> Now I have a couple of numpy related questions:
>
> In order to be able to use simd instructions I 
> create an ndarray subclass, which uses fftw_malloc to allocate the
> memory and fftw_free to free the memory when the array is deleted. This
> works fine for inplace operations however if someone does something like
> this:
>
> a = fftw3.AlignedArray(1024,complex)
>
> a = a+1
>
> a.ctypes.data points to a different memory location (this is actually an
> even bigger problem when executing fftw plans), however 
> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
>   

I can't comment about subclassing ndarrays, but I can give you a hint
about aligned allocator problem: you could maintain two list of cached
plans, automatically detect whether your arrays are aligned or not, and
use the appropriate list of plans; one list is for aligned arrays, one
for unaligned. Before removing support for fftw, I played with some C++
code to do exactly that. You can tell fftw to create plans for unaligned
arrays by using FFTW_UNALIGNED flag:

http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx

cheers,

David


From cycomanic at gmail.com  Mon Jan 26 23:29:05 2009
From: cycomanic at gmail.com (Jochen)
Date: Tue, 27 Jan 2009 17:29:05 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497E843D.6080005@ar.media.kyoto-u.ac.jp>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497E843D.6080005@ar.media.kyoto-u.ac.jp>
Message-ID: <1233030545.20296.57.camel@phy.auckland.ac.nz>

On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
> Jochen wrote:
> > Hi all,
> >
> > I just wrote ctypes bindings to fftw3 (see
> > http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> > for the post to scipy). 
> > Now I have a couple of numpy related questions:
> >
> > In order to be able to use simd instructions I 
> > create an ndarray subclass, which uses fftw_malloc to allocate the
> > memory and fftw_free to free the memory when the array is deleted. This
> > works fine for inplace operations however if someone does something like
> > this:
> >
> > a = fftw3.AlignedArray(1024,complex)
> >
> > a = a+1
> >
> > a.ctypes.data points to a different memory location (this is actually an
> > even bigger problem when executing fftw plans), however 
> > type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
> >   
> 
> I can't comment about subclassing ndarrays, but I can give you a hint
> about aligned allocator problem: you could maintain two list of cached
> plans, automatically detect whether your arrays are aligned or not, and
> use the appropriate list of plans; one list is for aligned arrays, one
> for unaligned. Before removing support for fftw, I played with some C++
> code to do exactly that. You can tell fftw to create plans for unaligned
> arrays by using FFTW_UNALIGNED flag:
> 
> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
> 
> cheers,
> 
> David

Hi David,

I have actually kept more closely to the fftw way of doing things, i.e.
I create a plan python object from two arrays, it also stores the two
arrays to prevent someone to delete the original arrays and then
executing the plan. You can then either execute the plan object, which
executes the fftw_plan, or you can call the fftw guru method
fftw_execute_dft(inarray, outarray), where I don't do any checking on
the arrays, to keep the performance high. I agree that such an approach
is not quite as intuitive as x=fft(y), but for my application
(propagation equations) I usually perform a large number of FFTs on two
arrays, so I just opted for staying close to the fftw way of doing
things. So my problem is not really so much with the alignment, it's
more with executing the plans, even if someone has created a plan from
two unaligned arrays, if he does something like x=x+1 somewhere in the
calculations and later executes a plan which was created using x, x will
not change because it's memory is not pointing to the original location.
I was looking for some way to prevent this. 

Cheers
Jochen
> ______________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From david at ar.media.kyoto-u.ac.jp  Mon Jan 26 23:28:43 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Jan 2009 13:28:43 +0900
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233030545.20296.57.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>	<497E843D.6080005@ar.media.kyoto-u.ac.jp>
	<1233030545.20296.57.camel@phy.auckland.ac.nz>
Message-ID: <497E8D7B.9080409@ar.media.kyoto-u.ac.jp>

Jochen wrote:
> On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
>   
>> Jochen wrote:
>>     
>>> Hi all,
>>>
>>> I just wrote ctypes bindings to fftw3 (see
>>> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
>>> for the post to scipy). 
>>> Now I have a couple of numpy related questions:
>>>
>>> In order to be able to use simd instructions I 
>>> create an ndarray subclass, which uses fftw_malloc to allocate the
>>> memory and fftw_free to free the memory when the array is deleted. This
>>> works fine for inplace operations however if someone does something like
>>> this:
>>>
>>> a = fftw3.AlignedArray(1024,complex)
>>>
>>> a = a+1
>>>
>>> a.ctypes.data points to a different memory location (this is actually an
>>> even bigger problem when executing fftw plans), however 
>>> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
>>>   
>>>       
>> I can't comment about subclassing ndarrays, but I can give you a hint
>> about aligned allocator problem: you could maintain two list of cached
>> plans, automatically detect whether your arrays are aligned or not, and
>> use the appropriate list of plans; one list is for aligned arrays, one
>> for unaligned. Before removing support for fftw, I played with some C++
>> code to do exactly that. You can tell fftw to create plans for unaligned
>> arrays by using FFTW_UNALIGNED flag:
>>
>> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
>>
>> cheers,
>>
>> David
>>     
>
> Hi David,
>
> I have actually kept more closely to the fftw way of doing things, i.e.
> I create a plan python object from two arrays, it also stores the two
> arrays to prevent someone to delete the original arrays and then
> executing the plan.

I am not sure I follow you when you say the "fftw way": you can avoid
having to store the arrays altogether while still using fftw plans,
there is nothing "unfftw" about using plans that way. I think trying to
guarantee that your arrays data buffers won't change is more complicated.

David


From cycomanic at gmail.com  Tue Jan 27 00:03:08 2009
From: cycomanic at gmail.com (Jochen)
Date: Tue, 27 Jan 2009 18:03:08 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497E8D7B.9080409@ar.media.kyoto-u.ac.jp>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497E843D.6080005@ar.media.kyoto-u.ac.jp>
	<1233030545.20296.57.camel@phy.auckland.ac.nz>
	<497E8D7B.9080409@ar.media.kyoto-u.ac.jp>
Message-ID: <1233032588.20296.82.camel@phy.auckland.ac.nz>

On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote:
> Jochen wrote:
> > On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
> >   
> >> Jochen wrote:
> >>     
> >>> Hi all,
> >>>
> >>> I just wrote ctypes bindings to fftw3 (see
> >>> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> >>> for the post to scipy). 
> >>> Now I have a couple of numpy related questions:
> >>>
> >>> In order to be able to use simd instructions I 
> >>> create an ndarray subclass, which uses fftw_malloc to allocate the
> >>> memory and fftw_free to free the memory when the array is deleted. This
> >>> works fine for inplace operations however if someone does something like
> >>> this:
> >>>
> >>> a = fftw3.AlignedArray(1024,complex)
> >>>
> >>> a = a+1
> >>>
> >>> a.ctypes.data points to a different memory location (this is actually an
> >>> even bigger problem when executing fftw plans), however 
> >>> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
> >>>   
> >>>       
> >> I can't comment about subclassing ndarrays, but I can give you a hint
> >> about aligned allocator problem: you could maintain two list of cached
> >> plans, automatically detect whether your arrays are aligned or not, and
> >> use the appropriate list of plans; one list is for aligned arrays, one
> >> for unaligned. Before removing support for fftw, I played with some C++
> >> code to do exactly that. You can tell fftw to create plans for unaligned
> >> arrays by using FFTW_UNALIGNED flag:
> >>
> >> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
> >>
> >> cheers,
> >>
> >> David
> >>     
> >
> > Hi David,
> >
> > I have actually kept more closely to the fftw way of doing things, i.e.
> > I create a plan python object from two arrays, it also stores the two
> > arrays to prevent someone to delete the original arrays and then
> > executing the plan.
> 
> I am not sure I follow you when you say the "fftw way": you can avoid
> having to store the arrays altogether while still using fftw plans,
> there is nothing "unfftw" about using plans that way. I think trying to
> guarantee that your arrays data buffers won't change is more complicated.
> 
> David

Sorry maybe I wasn't quite clear, what I mean by the "fftw way" is
creating a plan and executing the plan, instead of doing x=fft(y). As
you say the problem comes when you execute a plan on arrays which don't
exist anymore, which causes python to segfault (I'm talking about using
fftw_execute() not fftw_execute_dft). So yes essentially my problem is
trying to ensure that the buffer does not change. BTW memmap arrays have
the same problem
if I create a memmap array and later do something like 
a=a+1
all later changes will not be written to the file. 

BTW I just answered to you in the other thread on scipy-users as well,
I'll try to just keep the rest of my posts here so people don't have to
look at two lists if they want to follow things.


Cheers
Jochen

> ______
> _________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From david at ar.media.kyoto-u.ac.jp  Mon Jan 26 23:54:23 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Jan 2009 13:54:23 +0900
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233032588.20296.82.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>	<497E843D.6080005@ar.media.kyoto-u.ac.jp>	<1233030545.20296.57.camel@phy.auckland.ac.nz>	<497E8D7B.9080409@ar.media.kyoto-u.ac.jp>
	<1233032588.20296.82.camel@phy.auckland.ac.nz>
Message-ID: <497E937F.9020408@ar.media.kyoto-u.ac.jp>

Jochen wrote:
> On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote:
>   
>> Jochen wrote:
>>     
>>> On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
>>>   
>>>       
>>>> Jochen wrote:
>>>>     
>>>>         
>>>>> Hi all,
>>>>>
>>>>> I just wrote ctypes bindings to fftw3 (see
>>>>> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
>>>>> for the post to scipy). 
>>>>> Now I have a couple of numpy related questions:
>>>>>
>>>>> In order to be able to use simd instructions I 
>>>>> create an ndarray subclass, which uses fftw_malloc to allocate the
>>>>> memory and fftw_free to free the memory when the array is deleted. This
>>>>> works fine for inplace operations however if someone does something like
>>>>> this:
>>>>>
>>>>> a = fftw3.AlignedArray(1024,complex)
>>>>>
>>>>> a = a+1
>>>>>
>>>>> a.ctypes.data points to a different memory location (this is actually an
>>>>> even bigger problem when executing fftw plans), however 
>>>>> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
>>>>>   
>>>>>       
>>>>>           
>>>> I can't comment about subclassing ndarrays, but I can give you a hint
>>>> about aligned allocator problem: you could maintain two list of cached
>>>> plans, automatically detect whether your arrays are aligned or not, and
>>>> use the appropriate list of plans; one list is for aligned arrays, one
>>>> for unaligned. Before removing support for fftw, I played with some C++
>>>> code to do exactly that. You can tell fftw to create plans for unaligned
>>>> arrays by using FFTW_UNALIGNED flag:
>>>>
>>>> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
>>>>
>>>> cheers,
>>>>
>>>> David
>>>>     
>>>>         
>>> Hi David,
>>>
>>> I have actually kept more closely to the fftw way of doing things, i.e.
>>> I create a plan python object from two arrays, it also stores the two
>>> arrays to prevent someone to delete the original arrays and then
>>> executing the plan.
>>>       
>> I am not sure I follow you when you say the "fftw way": you can avoid
>> having to store the arrays altogether while still using fftw plans,
>> there is nothing "unfftw" about using plans that way. I think trying to
>> guarantee that your arrays data buffers won't change is more complicated.
>>
>> David
>>     
>
> Sorry maybe I wasn't quite clear, what I mean by the "fftw way" is
> creating a plan and executing the plan, instead of doing x=fft(y).

I guess I was not very clear, because my suggestion has nothing to do
with the above.

>  As
> you say the problem comes when you execute a plan on arrays which don't
> exist anymore, which causes python to segfault (I'm talking about using
> fftw_execute() not fftw_execute_dft). So yes essentially my problem is
> trying to ensure that the buffer does not change. 

I agree that if you just use fftw_execute, you will have segfaults: I am
just not sure that your problem is to ensure that the buffer does not
change :) You can instead handle relatively easily the case where the
buffers are not aligned, while still using fftw API. The fftw_execute is
just not an appropriate API for python code, IMHO.

Another solution would be to work on numpy itself, so that it use
aligned buffers, but that's obviously much longer term - that's just one
of those things on my TODO list that I never took the time to do,

David


From cycomanic at gmail.com  Tue Jan 27 00:43:29 2009
From: cycomanic at gmail.com (Jochen)
Date: Tue, 27 Jan 2009 18:43:29 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497E937F.9020408@ar.media.kyoto-u.ac.jp>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497E843D.6080005@ar.media.kyoto-u.ac.jp>
	<1233030545.20296.57.camel@phy.auckland.ac.nz>
	<497E8D7B.9080409@ar.media.kyoto-u.ac.jp>
	<1233032588.20296.82.camel@phy.auckland.ac.nz>
	<497E937F.9020408@ar.media.kyoto-u.ac.jp>
Message-ID: <1233035009.20296.98.camel@phy.auckland.ac.nz>

On Tue, 2009-01-27 at 13:54 +0900, David Cournapeau wrote:
> Jochen wrote:
> > On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote:
> >   
> >> Jochen wrote:
> >>     
> >>> On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
> >>>   
> >>>       
> >>>> Jochen wrote:
> >>>>     
> >>>>         
> >>>>> Hi all,
> >>>>>
> >>>>> I just wrote ctypes bindings to fftw3 (see
> >>>>> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> >>>>> for the post to scipy). 
> >>>>> Now I have a couple of numpy related questions:
> >>>>>
> >>>>> In order to be able to use simd instructions I 
> >>>>> create an ndarray subclass, which uses fftw_malloc to allocate the
> >>>>> memory and fftw_free to free the memory when the array is deleted. This
> >>>>> works fine for inplace operations however if someone does something like
> >>>>> this:
> >>>>>
> >>>>> a = fftw3.AlignedArray(1024,complex)
> >>>>>
> >>>>> a = a+1
> >>>>>
> >>>>> a.ctypes.data points to a different memory location (this is actually an
> >>>>> even bigger problem when executing fftw plans), however 
> >>>>> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
> >>>>>   
> >>>>>       
> >>>>>           
> >>>> I can't comment about subclassing ndarrays, but I can give you a hint
> >>>> about aligned allocator problem: you could maintain two list of cached
> >>>> plans, automatically detect whether your arrays are aligned or not, and
> >>>> use the appropriate list of plans; one list is for aligned arrays, one
> >>>> for unaligned. Before removing support for fftw, I played with some C++
> >>>> code to do exactly that. You can tell fftw to create plans for unaligned
> >>>> arrays by using FFTW_UNALIGNED flag:
> >>>>
> >>>> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
> >>>>
> >>>> cheers,
> >>>>
> >>>> David
> >>>>     
> >>>>         
> >>> Hi David,
> >>>
> >>> I have actually kept more closely to the fftw way of doing things, i.e.
> >>> I create a plan python object from two arrays, it also stores the two
> >>> arrays to prevent someone to delete the original arrays and then
> >>> executing the plan.
> >>>       
> >> I am not sure I follow you when you say the "fftw way": you can avoid
> >> having to store the arrays altogether while still using fftw plans,
> >> there is nothing "unfftw" about using plans that way. I think trying to
> >> guarantee that your arrays data buffers won't change is more complicated.
> >>
> >> David
> >>     
> >
> > Sorry maybe I wasn't quite clear, what I mean by the "fftw way" is
> > creating a plan and executing the plan, instead of doing x=fft(y).
> 
> I guess I was not very clear, because my suggestion has nothing to do
> with the above.
> 
> >  As
> > you say the problem comes when you execute a plan on arrays which don't
> > exist anymore, which causes python to segfault (I'm talking about using
> > fftw_execute() not fftw_execute_dft). So yes essentially my problem is
> > trying to ensure that the buffer does not change. 
> 
> I agree that if you just use fftw_execute, you will have segfaults: I am
> just not sure that your problem is to ensure that the buffer does not
> change :) You can instead handle relatively easily the case where the
> buffers are not aligned, while still using fftw API. The fftw_execute is
> just not an appropriate API for python code, IMHO.

Ah ok, I think I understand you now :). I agree that using fftw_execute
is not the most pythonic way of doing things. However every other way I
could think of, you would need to do a number of checks and possibly
create new plans, as well as keeping old plans around when doing a fft.
(I believe that's what you did in the old fftw bindings in scipy?). So I
decided to create fftw bindings first for maximum performance (I know,
just doing the whole calculation in c is probably more appropriate if
you want maximum performance ;), it also fits my needs. However I've
been thinking about ways to make the whole thing more intuitive. At
least now I can compare how much overhead I'm adding if I start to do
checks in order to make the handling easier :)

> 
> Another solution would be to work on numpy itself, so that it use
> aligned buffers, but that's obviously much longer term - that's just one
> of those things on my TODO list that I never took the time to do,
> 
I agree that would be nice, I'd think a couple of operations would
probably benefit from that. 

> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From david at ar.media.kyoto-u.ac.jp  Tue Jan 27 00:46:32 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Jan 2009 14:46:32 +0900
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233035009.20296.98.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>	<497E843D.6080005@ar.media.kyoto-u.ac.jp>	<1233030545.20296.57.camel@phy.auckland.ac.nz>	<497E8D7B.9080409@ar.media.kyoto-u.ac.jp>	<1233032588.20296.82.camel@phy.auckland.ac.nz>	<497E937F.9020408@ar.media.kyoto-u.ac.jp>
	<1233035009.20296.98.camel@phy.auckland.ac.nz>
Message-ID: <497E9FB8.5060900@ar.media.kyoto-u.ac.jp>

Jochen wrote:
> On Tue, 2009-01-27 at 13:54 +0900, David Cournapeau wrote:
>   
>> Jochen wrote:
>>     
>>> On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote:
>>>   
>>>       
>>>> Jochen wrote:
>>>>     
>>>>         
>>>>> On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Jochen wrote:
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I just wrote ctypes bindings to fftw3 (see
>>>>>>> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
>>>>>>> for the post to scipy). 
>>>>>>> Now I have a couple of numpy related questions:
>>>>>>>
>>>>>>> In order to be able to use simd instructions I 
>>>>>>> create an ndarray subclass, which uses fftw_malloc to allocate the
>>>>>>> memory and fftw_free to free the memory when the array is deleted. This
>>>>>>> works fine for inplace operations however if someone does something like
>>>>>>> this:
>>>>>>>
>>>>>>> a = fftw3.AlignedArray(1024,complex)
>>>>>>>
>>>>>>> a = a+1
>>>>>>>
>>>>>>> a.ctypes.data points to a different memory location (this is actually an
>>>>>>> even bigger problem when executing fftw plans), however 
>>>>>>> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
>>>>>>>   
>>>>>>>       
>>>>>>>           
>>>>>>>               
>>>>>> I can't comment about subclassing ndarrays, but I can give you a hint
>>>>>> about aligned allocator problem: you could maintain two list of cached
>>>>>> plans, automatically detect whether your arrays are aligned or not, and
>>>>>> use the appropriate list of plans; one list is for aligned arrays, one
>>>>>> for unaligned. Before removing support for fftw, I played with some C++
>>>>>> code to do exactly that. You can tell fftw to create plans for unaligned
>>>>>> arrays by using FFTW_UNALIGNED flag:
>>>>>>
>>>>>> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> David
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> Hi David,
>>>>>
>>>>> I have actually kept more closely to the fftw way of doing things, i.e.
>>>>> I create a plan python object from two arrays, it also stores the two
>>>>> arrays to prevent someone to delete the original arrays and then
>>>>> executing the plan.
>>>>>       
>>>>>           
>>>> I am not sure I follow you when you say the "fftw way": you can avoid
>>>> having to store the arrays altogether while still using fftw plans,
>>>> there is nothing "unfftw" about using plans that way. I think trying to
>>>> guarantee that your arrays data buffers won't change is more complicated.
>>>>
>>>> David
>>>>     
>>>>         
>>> Sorry maybe I wasn't quite clear, what I mean by the "fftw way" is
>>> creating a plan and executing the plan, instead of doing x=fft(y).
>>>       
>> I guess I was not very clear, because my suggestion has nothing to do
>> with the above.
>>
>>     
>>>  As
>>> you say the problem comes when you execute a plan on arrays which don't
>>> exist anymore, which causes python to segfault (I'm talking about using
>>> fftw_execute() not fftw_execute_dft). So yes essentially my problem is
>>> trying to ensure that the buffer does not change. 
>>>       
>> I agree that if you just use fftw_execute, you will have segfaults: I am
>> just not sure that your problem is to ensure that the buffer does not
>> change :) You can instead handle relatively easily the case where the
>> buffers are not aligned, while still using fftw API. The fftw_execute is
>> just not an appropriate API for python code, IMHO.
>>     
>
> Ah ok, I think I understand you now :). I agree that using fftw_execute
> is not the most pythonic way of doing things. However every other way I
> could think of, you would need to do a number of checks and possibly
> create new plans, as well as keeping old plans around when doing a fft.
>   

Yes, but I don't see the problem with keeping old plans/creating new
ones. It was a pain in C++, because I had to handle many backends, not
just fftw, but in your case, doing it in python is not very difficult I
think.

> (I believe that's what you did in the old fftw bindings in scipy?).

that's what was done in scipy.fftpack originally, but not by me.

>  So I
> decided to create fftw bindings first for maximum performance (I know,
> just doing the whole calculation in c is probably more appropriate if
> you want maximum performance ;), it also fits my needs.

Caching plans will not affect performances; on the contrary, you will be
able to use more aggressively optimized plans if you add the ability to
save/load plans to the disk (using wisdow mechanism). Checking for
aligned arrays is one line in python, and should not affect much
performances, specially if you do fft on big arrays,

David


From cycomanic at gmail.com  Tue Jan 27 01:31:37 2009
From: cycomanic at gmail.com (Jochen)
Date: Tue, 27 Jan 2009 19:31:37 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497E9FB8.5060900@ar.media.kyoto-u.ac.jp>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497E843D.6080005@ar.media.kyoto-u.ac.jp>
	<1233030545.20296.57.camel@phy.auckland.ac.nz>
	<497E8D7B.9080409@ar.media.kyoto-u.ac.jp>
	<1233032588.20296.82.camel@phy.auckland.ac.nz>
	<497E937F.9020408@ar.media.kyoto-u.ac.jp>
	<1233035009.20296.98.camel@phy.auckland.ac.nz>
	<497E9FB8.5060900@ar.media.kyoto-u.ac.jp>
Message-ID: <1233037897.20296.109.camel@phy.auckland.ac.nz>

On Tue, 2009-01-27 at 14:46 +0900, David Cournapeau wrote:
> Jochen wrote:
> > On Tue, 2009-01-27 at 13:54 +0900, David Cournapeau wrote:
> >   
> >> Jochen wrote:
> >>     
> >>> On Tue, 2009-01-27 at 13:28 +0900, David Cournapeau wrote:
> >>>   
> >>>       
> >>>> Jochen wrote:
> >>>>     
> >>>>         
> >>>>> On Tue, 2009-01-27 at 12:49 +0900, David Cournapeau wrote:
> >>>>>   
> >>>>>       
> >>>>>           
> >>>>>> Jochen wrote:
> >>>>>>     
> >>>>>>         
> >>>>>>             
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I just wrote ctypes bindings to fftw3 (see
> >>>>>>> http://projects.scipy.org/pipermail/scipy-user/2009-January/019557.html
> >>>>>>> for the post to scipy). 
> >>>>>>> Now I have a couple of numpy related questions:
> >>>>>>>
> >>>>>>> In order to be able to use simd instructions I 
> >>>>>>> create an ndarray subclass, which uses fftw_malloc to allocate the
> >>>>>>> memory and fftw_free to free the memory when the array is deleted. This
> >>>>>>> works fine for inplace operations however if someone does something like
> >>>>>>> this:
> >>>>>>>
> >>>>>>> a = fftw3.AlignedArray(1024,complex)
> >>>>>>>
> >>>>>>> a = a+1
> >>>>>>>
> >>>>>>> a.ctypes.data points to a different memory location (this is actually an
> >>>>>>> even bigger problem when executing fftw plans), however 
> >>>>>>> type(a) still gives me <class 'fftw3.planning.AlignedArray'>.
> >>>>>>>   
> >>>>>>>       
> >>>>>>>           
> >>>>>>>               
> >>>>>> I can't comment about subclassing ndarrays, but I can give you a hint
> >>>>>> about aligned allocator problem: you could maintain two list of cached
> >>>>>> plans, automatically detect whether your arrays are aligned or not, and
> >>>>>> use the appropriate list of plans; one list is for aligned arrays, one
> >>>>>> for unaligned. Before removing support for fftw, I played with some C++
> >>>>>> code to do exactly that. You can tell fftw to create plans for unaligned
> >>>>>> arrays by using FFTW_UNALIGNED flag:
> >>>>>>
> >>>>>> http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/backends/fftw3/src/zfft.cxx
> >>>>>>
> >>>>>> cheers,
> >>>>>>
> >>>>>> David
> >>>>>>     
> >>>>>>         
> >>>>>>             
> >>>>> Hi David,
> >>>>>
> >>>>> I have actually kept more closely to the fftw way of doing things, i.e.
> >>>>> I create a plan python object from two arrays, it also stores the two
> >>>>> arrays to prevent someone to delete the original arrays and then
> >>>>> executing the plan.
> >>>>>       
> >>>>>           
> >>>> I am not sure I follow you when you say the "fftw way": you can avoid
> >>>> having to store the arrays altogether while still using fftw plans,
> >>>> there is nothing "unfftw" about using plans that way. I think trying to
> >>>> guarantee that your arrays data buffers won't change is more complicated.
> >>>>
> >>>> David
> >>>>     
> >>>>         
> >>> Sorry maybe I wasn't quite clear, what I mean by the "fftw way" is
> >>> creating a plan and executing the plan, instead of doing x=fft(y).
> >>>       
> >> I guess I was not very clear, because my suggestion has nothing to do
> >> with the above.
> >>
> >>     
> >>>  As
> >>> you say the problem comes when you execute a plan on arrays which don't
> >>> exist anymore, which causes python to segfault (I'm talking about using
> >>> fftw_execute() not fftw_execute_dft). So yes essentially my problem is
> >>> trying to ensure that the buffer does not change. 
> >>>       
> >> I agree that if you just use fftw_execute, you will have segfaults: I am
> >> just not sure that your problem is to ensure that the buffer does not
> >> change :) You can instead handle relatively easily the case where the
> >> buffers are not aligned, while still using fftw API. The fftw_execute is
> >> just not an appropriate API for python code, IMHO.
> >>     
> >
> > Ah ok, I think I understand you now :). I agree that using fftw_execute
> > is not the most pythonic way of doing things. However every other way I
> > could think of, you would need to do a number of checks and possibly
> > create new plans, as well as keeping old plans around when doing a fft.
> >   
> 
> Yes, but I don't see the problem with keeping old plans/creating new
> ones. It was a pain in C++, because I had to handle many backends, not
> just fftw, but in your case, doing it in python is not very difficult I
> think.
> 
> > (I believe that's what you did in the old fftw bindings in scipy?).
> 
> that's what was done in scipy.fftpack originally, but not by me.
> 
> >  So I
> > decided to create fftw bindings first for maximum performance (I know,
> > just doing the whole calculation in c is probably more appropriate if
> > you want maximum performance ;), it also fits my needs.
> 
> Caching plans will not affect performances; on the contrary, you will be
> able to use more aggressively optimized plans if you add the ability to
> save/load plans to the disk (using wisdow mechanism). Checking for
> aligned arrays is one line in python, and should not affect much
> performances, specially if you do fft on big arrays,
> 
I agree checking for aligned arrays does not affect performance much, I
guess the lookup mechanism for the plans is what would take up the most
time IMO. 
I'll definitely look into this though :)

Cheers
Jochen




From sturla at molden.no  Tue Jan 27 06:04:46 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 27 Jan 2009 12:04:46 +0100
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233032588.20296.82.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>	<497E843D.6080005@ar.media.kyoto-u.ac.jp>	<1233030545.20296.57.camel@phy.auckland.ac.nz>	<497E8D7B.9080409@ar.media.kyoto-u.ac.jp>
	<1233032588.20296.82.camel@phy.auckland.ac.nz>
Message-ID: <497EEA4E.9030009@molden.no>

On 1/27/2009 6:03 AM, Jochen wrote:

 > BTW memmap arrays have
> the same problem
> if I create a memmap array and later do something like 
> a=a+1
> all later changes will not be written to the file. 

= is Python's rebinding operator.

a = a + 1 rebinds a to a different object.

As for ndarray's, I'd like to point out the difference between

a[:] = a + 1

and

a = a + 1



S.M.






From sturla at molden.no  Tue Jan 27 06:37:45 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 27 Jan 2009 12:37:45 +0100
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233015981.4180.39.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
Message-ID: <497EF209.1010403@molden.no>

On 1/27/2009 1:26 AM, Jochen wrote:

> a = fftw3.AlignedArray(1024,complex)
> 
> a = a+1

= used this way is not assignment, it is name binding.

It is easy to use function's like fftw_malloc with NumPy:


import ctypes
import numpy

fftw_malloc = ctypes.cdll.fftw.fftw_malloc
fftw_malloc.argtypes = [ctypes.c_ulong,]
fftw_malloc.restype = ctypes.c_ulong

def aligned_array(N, dtype):
     d = dtype()
     address = fftw_malloc(N * d.nbytes) # restype = ctypes.c_ulong
     if (address = 0): raise MemoryError, 'fftw_malloc returned NULL'
     class Dummy(object): pass
     d = Dummy()
     d.__array_interface__ = {
         'data' = (address, False),
         'typestr' : dtype.str,
         'descr' : dtype.descr,
         'shape' : shape,
         'strides' : strides,
         'version' : 3
     }
     return numpy.asarray(d)


If you have to check for a particular alignment before calling fftw, 
that is trivial as well:


def is_aligned(array, boundary):
     address = array.__array_interface__['data'][0]
     return not(address % boundary)




> there a way that I get a different object type?  Or even better is there
> a way to prevent operations like a=a+1 or make them automatically
> in-place operations?

a = a + 1 # rebinds the name 'a' to another array

a[:] = a + 1 # fills a with the result of a + 1


This has to do with Python syntax, not NumPy per se. You cannot overload 
the behaviour of Python's name binding operator (=).


Sturla Molden





From sturla at molden.no  Tue Jan 27 06:40:09 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 27 Jan 2009 12:40:09 +0100
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497EF209.1010403@molden.no>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497EF209.1010403@molden.no>
Message-ID: <497EF299.6040705@molden.no>

On 1/27/2009 12:37 PM, Sturla Molden wrote:

>      address = fftw_malloc(N * d.nbytes) # restype = ctypes.c_ulong
>      if (address = 0): 
        if (address == ): raise MemoryError, 'fftw_malloc returned NULL'


Sorry for the typo.


S.M.



From sturla at molden.no  Tue Jan 27 07:07:21 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 27 Jan 2009 13:07:21 +0100
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497EF209.1010403@molden.no>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497EF209.1010403@molden.no>
Message-ID: <497EF8F9.2070106@molden.no>

On 1/27/2009 12:37 PM, Sturla Molden wrote:

> It is easy to use function's like fftw_malloc with NumPy:

Besides this, if I were to write a wrapper for FFTW in Python, I would 
consider wrapping FFTW's Fortran interface with f2py.

It is probably safer, as well as faster, than using ctypes. It would 
also allow the FFTW library to be linked statically to the Python 
extension, avoiding DLL hell.

S.M.



From klemm at phys.ethz.ch  Tue Jan 27 07:24:38 2009
From: klemm at phys.ethz.ch (Hanno Klemm)
Date: Tue, 27 Jan 2009 13:24:38 +0100
Subject: [Numpy-discussion] Slicing and structured arrays question
Message-ID: <twig.1233059078.18288@phys.ethz.ch>


Hi, 
I have the following question, that I could not find an answer to in the
example list, or by googling:

I have a record array with dtype such as:
dtype([('times', '<f8'), ('sensors', '|S8'), ('prop1', '<f8'), ('prop2',
'<f8'), ('prop3', '<f8'), ('prop4', '<f8')])

I would now like to calculate the mean and median for each of the
properties 'props'. Is there a way to do this similarly to a conventional
array with

a[:,2:].mean(axis=0)

or do I have to use a loop over the names of the properties? 

Thanks in advance,
Hanno







-- 
Hanno Klemm
klemm at phys.ethz.ch




From sturla at molden.no  Tue Jan 27 08:16:32 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 27 Jan 2009 14:16:32 +0100
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497EF209.1010403@molden.no>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497EF209.1010403@molden.no>
Message-ID: <497F0930.2000202@molden.no>

On 1/27/2009 12:37 PM, Sturla Molden wrote:

> import ctypes
> import numpy
> 
> fftw_malloc = ctypes.cdll.fftw.fftw_malloc
> fftw_malloc.argtypes = [ctypes.c_ulong,]
> fftw_malloc.restype = ctypes.c_ulong
> 
> def aligned_array(N, dtype):
>      d = dtype()
>      address = fftw_malloc(N * d.nbytes) # restype = ctypes.c_ulong
>      if (address = 0): raise MemoryError, 'fftw_malloc returned NULL'
>      class Dummy(object): pass
>      d = Dummy()
>      d.__array_interface__ = {
>          'data' = (address, False),
>          'typestr' : dtype.str,
>          'descr' : dtype.descr,
>          'shape' : shape,
>          'strides' : strides,
>          'version' : 3
>      }
>      return numpy.asarray(d)


Or if you just want to use numpy, aligning to a 16 byte boundary
can be done like this:


def aligned_array(N, dtype):
     d = dtype()
     tmp = numpy.array(N * d.nbytes + 16, dtype=numpy.uint8)
     address = tmp.__array_interface__['data'][0]
     offset = (16 - address % 16) % 16
     return tmp[offset:offset+N].view(dtype=dtype)


S.M.

















From hanni.ali at gmail.com  Tue Jan 27 09:45:51 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Tue, 27 Jan 2009 14:45:51 +0000
Subject: [Numpy-discussion] PyArray_Zeros
Message-ID: <789d27b10901270645j6992b680v8beca876b3a761a2@mail.gmail.com>

Hi,

I have been having trouble with the PyArray_Zeros/PyArray_ZEROS functions. I
cannot seem to create an array using these functions.

resultArray = PyArray_ZEROS(otherArray->nd, otherArray->dimensions,
NPY_DOUBLE, 0);

I would have thought this would have created an array the same shape as the
otherArray, just filled with zeros...

But I seem to get an error.

Am I doing something obviously wrong?

Hanni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090127/a580aeb8/attachment.html>

From oliphant at enthought.com  Tue Jan 27 10:06:52 2009
From: oliphant at enthought.com (Travis Oliphant)
Date: Tue, 27 Jan 2009 09:06:52 -0600
Subject: [Numpy-discussion] Slicing and structured arrays question
In-Reply-To: <twig.1233059078.18288@phys.ethz.ch>
References: <twig.1233059078.18288@phys.ethz.ch>
Message-ID: <497F230C.8030401@enthought.com>

Hanno Klemm wrote:
> Hi, 
> I have the following question, that I could not find an answer to in the
> example list, or by googling:
>
> I have a record array with dtype such as:
> dtype([('times', '<f8'), ('sensors', '|S8'), ('prop1', '<f8'), ('prop2',
> '<f8'), ('prop3', '<f8'), ('prop4', '<f8')])
>
> I would now like to calculate the mean and median for each of the
> properties 'props'. Is there a way to do this similarly to a conventional
> array with
>
> a[:,2:].mean(axis=0)
>
> or do I have to use a loop over the names of the properties? 
>   
You must first view the array as a float array in order to calculate the 
mean:

This should work:

b = a.view(float)
b.shape = (-1,6)
b[:,2:].mean(axis=0)


-Travis



From cycomanic at gmail.com  Tue Jan 27 14:19:39 2009
From: cycomanic at gmail.com (Jochen)
Date: Wed, 28 Jan 2009 08:19:39 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497EF209.1010403@molden.no>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497EF209.1010403@molden.no>
Message-ID: <1233083979.20296.122.camel@phy.auckland.ac.nz>

On Tue, 2009-01-27 at 12:37 +0100, Sturla Molden wrote:
> On 1/27/2009 1:26 AM, Jochen wrote:
> 
> > a = fftw3.AlignedArray(1024,complex)
> > 
> > a = a+1
> 
> = used this way is not assignment, it is name binding.
> 
> It is easy to use function's like fftw_malloc with NumPy:
> 
> 
> import ctypes
> import numpy
> 
> fftw_malloc = ctypes.cdll.fftw.fftw_malloc
> fftw_malloc.argtypes = [ctypes.c_ulong,]
> fftw_malloc.restype = ctypes.c_ulong
> 
> def aligned_array(N, dtype):
>      d = dtype()
>      address = fftw_malloc(N * d.nbytes) # restype = ctypes.c_ulong
>      if (address = 0): raise MemoryError, 'fftw_malloc returned NULL'
>      class Dummy(object): pass
>      d = Dummy()
>      d.__array_interface__ = {
>          'data' = (address, False),
>          'typestr' : dtype.str,
>          'descr' : dtype.descr,
>          'shape' : shape,
>          'strides' : strides,
>          'version' : 3
>      }
>      return numpy.asarray(d)
> 

I actually do it slightly different, because I want to free the memory
using fftw_free. So I'm subclassing ndarray and in the __new__ function
of the subclass I create a buffer object from fftw_malloc using
PyBuffer_FromReadWriteMemory and pass that to ndarray.__new__. In
__del__ I check if the .base is a buffer object and do an fftw_free.

> 
> If you have to check for a particular alignment before calling fftw, 
> that is trivial as well:
> 
> 
> def is_aligned(array, boundary):
>      address = array.__array_interface__['data'][0]
>      return not(address % boundary)
> 
> 

Yes I knew that, btw do you know of any systems which need alignment to
boundaries other than 16byte for simd operations?

> 
> 
> > there a way that I get a different object type?  Or even better is there
> > a way to prevent operations like a=a+1 or make them automatically
> > in-place operations?
> 
> a = a + 1 # rebinds the name 'a' to another array
> 
> a[:] = a + 1 # fills a with the result of a + 1
>
I knew about this one, this is what I have been doing.
> 
> This has to do with Python syntax, not NumPy per se. You cannot overload 
> the behaviour of Python's name binding operator (=).
> 
Yes I actually thought about it some more yesterday when going home and
realised that this would really not be possible. I guess I just have to
document it clearly so that people don't run into it.

Cheers
Jochen
> 
> Sturla Molden
> 
> 
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From cycomanic at gmail.com  Tue Jan 27 14:23:44 2009
From: cycomanic at gmail.com (Jochen)
Date: Wed, 28 Jan 2009 08:23:44 +1300
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <497F0930.2000202@molden.no>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497EF209.1010403@molden.no>  <497F0930.2000202@molden.no>
Message-ID: <1233084224.20296.125.camel@phy.auckland.ac.nz>

On Tue, 2009-01-27 at 14:16 +0100, Sturla Molden wrote:
> On 1/27/2009 12:37 PM, Sturla Molden wrote:
> 
> > import ctypes
> > import numpy
> > 
> > fftw_malloc = ctypes.cdll.fftw.fftw_malloc
> > fftw_malloc.argtypes = [ctypes.c_ulong,]
> > fftw_malloc.restype = ctypes.c_ulong
> > 
> > def aligned_array(N, dtype):
> >      d = dtype()
> >      address = fftw_malloc(N * d.nbytes) # restype = ctypes.c_ulong
> >      if (address = 0): raise MemoryError, 'fftw_malloc returned NULL'
> >      class Dummy(object): pass
> >      d = Dummy()
> >      d.__array_interface__ = {
> >          'data' = (address, False),
> >          'typestr' : dtype.str,
> >          'descr' : dtype.descr,
> >          'shape' : shape,
> >          'strides' : strides,
> >          'version' : 3
> >      }
> >      return numpy.asarray(d)
> 
> 
> Or if you just want to use numpy, aligning to a 16 byte boundary
> can be done like this:
> 
> 
> def aligned_array(N, dtype):
>      d = dtype()
>      tmp = numpy.array(N * d.nbytes + 16, dtype=numpy.uint8)
>      address = tmp.__array_interface__['data'][0]
>      offset = (16 - address % 16) % 16
>      return tmp[offset:offset+N].view(dtype=dtype)
> 
> 
> S.M.
> 
Ah, I didn't think about doing it in python, cool thanks.


> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From rmay31 at gmail.com  Tue Jan 27 14:37:51 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 27 Jan 2009 13:37:51 -0600
Subject: [Numpy-discussion] recfunctions.stack_arrays
Message-ID: <497F628F.1020604@gmail.com>

Pierre (or anyone else who cares to chime in),

I'm using stack_arrays to combine data from two different files into a single
array.  In one of these files, the data from one entire record comes back
missing, which, thanks to your recent change, ends up having a boolean dtype.
There is actual data for this same field in the 2nd file, so it ends up having
the dtype of float64.  When I try to combine the two arrays, I end up with the
following traceback:

    data = stack_arrays((old_data, data))
  File "/home/rmay/.local/lib64/python2.5/site-packages/metpy/cbook.py", line
260, in stack_arrays
    output = ma.masked_all((np.sum(nrecords),), newdescr)
  File "/home/rmay/.local/lib64/python2.5/site-packages/numpy/ma/extras.py", line
79, in masked_all
    a = masked_array(np.empty(shape, dtype),
ValueError: two fields with the same name

Which is unsurprising.  Do you think there is any reasonable way to get
stack_arrays() to find a common dtype for fields with the same name?  Or another
suggestion on how to approach this?  If you think coercing one/both of the fields
to a common dtype is the way to go, just point me to a function that could figure
out the dtype and I'll try to put together a patch.

Thanks,

Ryan

P.S.  Thanks so much for your work on putting those utility functions in
recfunctions.py  It makes it so much easier to have these functions available in
the library itself rather than needing to reinvent the wheel over and over.

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From pgmdevlist at gmail.com  Tue Jan 27 15:03:39 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 27 Jan 2009 15:03:39 -0500
Subject: [Numpy-discussion] recfunctions.stack_arrays
In-Reply-To: <497F628F.1020604@gmail.com>
References: <497F628F.1020604@gmail.com>
Message-ID: <4BC6B814-42C8-4695-A14F-E14A02F504F4@gmail.com>

[Some background: we're talking about numpy.lib.recfunctions, a set of  
functions to manipulate structured arrays]

Ryan,
If the two files have the same structure, you can use that fact and  
specify the dtype of the output directly with the dtype parameter of  
mafromtxt. That way, you're sure that the two arrays will have the  
same dtype. If you don't know the structure beforehand, you could try  
to load one array and use its dtype as input of mafromtxt to load the  
second one.
Now, we could also try to modify stack_arrays so that it would take  
the largest dtype when several fields have the same name. I'm not  
completely satisfied by this approach, as it makes dtype conversions  
under the hood. Maybe we could provide the functionality as an option  
(w/ a forced_conversion boolean input parameter) ?
I'm a bit surprised by the error message you get. If I try:

 >>> a = ma.array([(1,2,3)], mask=[(0,1,0)], dtype=[('a',int),  
('b',bool), ('c',float)])
 >>> b = ma.array([(4, 5, 6)], dtype=[('a', int), ('b', float), ('c',  
float)])
 >>> test = np.stack_arrays((a, b))

I get a TypeError instead (the field 'b' hasn't the same type in a and  
b). Now, I get the 'two fields w/ the same name' when I use  
np.merge_arrays (with the flatten option). Could you send a small  
example ?

>
> P.S.  Thanks so much for your work on putting those utility  
> functions in
> recfunctions.py  It makes it so much easier to have these functions  
> available in
> the library itself rather than needing to reinvent the wheel over  
> and over.

Indeed. Note that most of the job had been done by John Hunter and the  
matplotlib developer in their matplotlib.mlab module, so you should  
thank them and not me. I just cleaned up some of the functions.


From nwagner at iam.uni-stuttgart.de  Tue Jan 27 15:11:56 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 27 Jan 2009 21:11:56 +0100
Subject: [Numpy-discussion] make latex in numpy/doc failed
Message-ID: <web-116379466@uni-stuttgart.de>

Hi all,

a make latex in numpy/doc failed with

...

Intersphinx hit: PyObject 
http://docs.python.org/dev/c-api/structures.html
writing... Sphinx error:
too many nesting section levels for LaTeX, at heading: 
numpy.ma.MaskedArray.__lt__
make: *** [latex] Fehler 1
  

I am using sphinxv0.5.1
BTW, make html works fine here.

Nils


From sturla at molden.no  Tue Jan 27 15:26:48 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 27 Jan 2009 21:26:48 +0100 (CET)
Subject: [Numpy-discussion] numpy ndarray questions
In-Reply-To: <1233084224.20296.125.camel@phy.auckland.ac.nz>
References: <1233015981.4180.39.camel@phy.auckland.ac.nz>
	<497EF209.1010403@molden.no> <497F0930.2000202@molden.no>
	<1233084224.20296.125.camel@phy.auckland.ac.nz>
Message-ID: <3d8df38227c4312367d440c39f67c0de.squirrel@webmail.uio.no>

> On Tue, 2009-01-27 at 14:16 +0100, Sturla Molden wrote:

>> def aligned_array(N, dtype):
>>      d = dtype()
>>      tmp = numpy.array(N * d.nbytes + 16, dtype=numpy.uint8)
>>      address = tmp.__array_interface__['data'][0]
>>      offset = (16 - address % 16) % 16
>>      return tmp[offset:offset+N].view(dtype=dtype)

> Ah, I didn't think about doing it in python, cool thanks.


Doing it from Python means you don't have to worry about manually
deallocating the array afterwards.

It seems the issue of 16-byte alignment has to do with efficient data
alignment for SIMD instructions (SSE, MMX, etc). So this is not just an
FFTW issue.

I would just put a check for 16-byte alignment in the wrapper, and raise
an exception (e.g. MemoryError) if the array is not aligned properly.
Raising an exception will inform the user of the problem. I would not
attempt to make a local copy if the array is errorneously aligned. That is
my personal preference.

Sturla Molden




From rmay31 at gmail.com  Tue Jan 27 16:23:35 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 27 Jan 2009 15:23:35 -0600
Subject: [Numpy-discussion] recfunctions.stack_arrays
In-Reply-To: <4BC6B814-42C8-4695-A14F-E14A02F504F4@gmail.com>
References: <497F628F.1020604@gmail.com>
	<4BC6B814-42C8-4695-A14F-E14A02F504F4@gmail.com>
Message-ID: <497F7B57.3020709@gmail.com>

Pierre GM wrote:
> [Some background: we're talking about numpy.lib.recfunctions, a set of  
> functions to manipulate structured arrays]
> 
> Ryan,
> If the two files have the same structure, you can use that fact and  
> specify the dtype of the output directly with the dtype parameter of  
> mafromtxt. That way, you're sure that the two arrays will have the  
> same dtype. If you don't know the structure beforehand, you could try  
> to load one array and use its dtype as input of mafromtxt to load the  
> second one.

I could force the dtype.  However, since the flexibility is there in mafromtxt,
I'd like to avoid hard coding the dtype, so I don't have to worry about updating
the code if the file format ever changes (this parses live data).

> Now, we could also try to modify stack_arrays so that it would take  
> the largest dtype when several fields have the same name. I'm not  
> completely satisfied by this approach, as it makes dtype conversions  
> under the hood. Maybe we could provide the functionality as an option  
> (w/ a forced_conversion boolean input parameter) ?

I definitely wouldn't advocate magic by default, but I think it would be nice to
be able to get the functionality if one wanted to.  There is one problem I
noticed, however.  I found common_type and lib.mintypecode, but both raise errors
 when trying to find a dtype to match both bool and float.  I don't know if
there's another function somewhere that would work for what I want.

> I'm a bit surprised by the error message you get. If I try:
> 
>  >>> a = ma.array([(1,2,3)], mask=[(0,1,0)], dtype=[('a',int),  
> ('b',bool), ('c',float)])
>  >>> b = ma.array([(4, 5, 6)], dtype=[('a', int), ('b', float), ('c',  
> float)])
>  >>> test = np.stack_arrays((a, b))
> 
> I get a TypeError instead (the field 'b' hasn't the same type in a and  
> b). Now, I get the 'two fields w/ the same name' when I use  
> np.merge_arrays (with the flatten option). Could you send a small  
> example ?

Apparently, I get my error as a result of my use of titles in the dtype to store
an alternate name for the field.  (If you're not familiar with titles, they're
nice because you can get fields by either name, so for the following example,
a['a'] and a['A'] both return array([1]).)  The following version of your case
gives me the ValueError:

>>> from numpy.lib.recfunctions import stack_arrays
>>> a = ma.array([(1,2,3)], mask=[(0,1,0)], dtype=[(('a','A'),int),
(('b','B'),bool), (('c','C'),float)])
>>> b = ma.array([(4,5,6)], dtype=[(('a','A'),int), (('b','B'),float),
(('c','C'),float)])
>>> stack_arrays((a,b))
ValueError: two fields with the same name

As a side question, do you have some local mods to your numpy SVN so that some of
the functions in recfunctions are available in numpy's top level?  On mine, I
can't get to them except by importing them from numpy.lib.recfunctions.  I don't
see any mention of recfunctions in lib/__init__.py.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From pgmdevlist at gmail.com  Tue Jan 27 17:00:42 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 27 Jan 2009 17:00:42 -0500
Subject: [Numpy-discussion] recfunctions.stack_arrays
In-Reply-To: <497F7B57.3020709@gmail.com>
References: <497F628F.1020604@gmail.com>
	<4BC6B814-42C8-4695-A14F-E14A02F504F4@gmail.com>
	<497F7B57.3020709@gmail.com>
Message-ID: <290D06CA-57FD-46B2-B653-82822620EB3E@gmail.com>


On Jan 27, 2009, at 4:23 PM, Ryan May wrote:

>
> I definitely wouldn't advocate magic by default, but I think it  
> would be nice to
> be able to get the functionality if one wanted to.

OK. Put on the TODO list.


> There is one problem I
> noticed, however.  I found common_type and lib.mintypecode, but both  
> raise errors
> when trying to find a dtype to match both bool and float.  I don't  
> know if
> there's another function somewhere that would work for what I want.

I'm not familiar with these functions, I'll check that.
>
> Apparently, I get my error as a result of my use of titles in the  
> dtype to store
> an alternate name for the field.  (If you're not familiar with  
> titles, they're
> nice because you can get fields by either name, so for the following  
> example,
> a['a'] and a['A'] both return array([1]).)  The following version of  
> your case
> gives me the ValueError:

Ah OK. You found a bug. There's a frustrating feature of dtypes:  
dtype.names doesn't always match [_[0] for _ in dtype.descr].


> As a side question, do you have some local mods to your numpy SVN so  
> that some of
> the functions in recfunctions are available in numpy's top level?

Probably. I used the develop option of setuptools to install numpy on  
a virtual environment.

> On mine, I
> can't get to them except by importing them from  
> numpy.lib.recfunctions.  I don't
> see any mention of recfunctions in lib/__init__.py.


Well, till some problems are ironed out, I'm not really in favor of  
advertising them too much...



From mcolonno at gmail.com  Tue Jan 27 20:05:07 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Tue, 27 Jan 2009 17:05:07 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
Message-ID: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>

   Hi ~

   I'm trying to build numpy (hopefully eventually scipy with the same
setup) with the Intel compilers (and Intel MKL) on the WinXP 64-bit
platform. Finding / linking to the Intel MKL seems to be successful (see
below) but I have an issue with the settings defined somewhere in the
various setup scripts (can't find where). Per the output below, the Intel
compilers on Windows are looking for ".obj" object files rather than the
Linux-style ".o" files. I'd also like to get rid of the /L and -L flags (no
longer supported in Intel C++ v. 11.0 it seems) but this just throws a
warning and does not seem to cause any problems. Can anyone point me to the
python file(s) I need to edit to modify the .o object file setting to .obj?
(The file "_configtest.obj" is created.) Once operational, I'll pass along
all of my config info for anyone else building in this environment.

   Thanks!
   ~Mike C.


Output from build:

>>python setup.py config --compiler=intel --fcompiler=intelem install

Running from numpy source directory.
Forcing DISTUTILS_USE_SDK=1
F2PY Version 2_5972
blas_opt_info:
blas_mkl_info:
  FOUND:
    libraries = ['mkl_em64t', 'mkl_dll']
    library_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\
061\\cpp\\m
kl\\em64t\\lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\include']

  FOUND:
    libraries = ['mkl_em64t', 'mkl_dll']
    library_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\em64t\\lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\include']

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  FOUND:
    libraries = ['mkl_em64t', 'mkl_dll']
    library_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\em64t\\lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\include']

  FOUND:
    libraries = ['mkl_lapack', 'mkl_em64t', 'mkl_dll']
    library_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\em64t\\lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\include']

  FOUND:
    libraries = ['mkl_lapack', 'mkl_em64t', 'mkl_dll']
    library_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\em64t\\lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['C:\\Program Files
(x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m
kl\\include']

running config
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler
opti
ons
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
opt
ions
running build_src
building py_modules sources
creating build
creating build\src.win32-2.5
creating build\src.win32-2.5\numpy
creating build\src.win32-2.5\numpy\distutils
building extension "numpy.core.multiarray" sources
creating build\src.win32-2.5\numpy\core
Generating build\src.win32-2.5\numpy\core\include/numpy\config.h
Could not locate executable icc
Could not locate executable ecc
Ignoring "MSVCCompiler instance has no attribute '_MSVCCompiler__root'" (I
think
 it is msvccompiler.py bug)
customize IntelEM64TFCompiler
Found executable C:\Program Files
(x86)\Intel\Compiler\11.0\061\fortran\Bin\inte
l64\ifort.exe
Found executable C:\Program Files
(x86)\Intel\Compiler\11.0\061\fortran\Bin\inte
l64\ifort.exe
C compiler: icl

compile options: '-IC:\Python25\include -Inumpy\core\src
-Inumpy\core\include -I
C:\Python25\include -IC:\Python25\PC -c'
icl: _configtest.c
Found executable C:\Program Files
(x86)\Intel\Compiler\11.0\061\cpp\Bin\intel64\
icl.exe
icl _configtest.o -LC:\Python25\lib -LC:\ -LC:\Python25\libs -o _configtest
Intel(R) C++ Intel(R) 64 Compiler Professional for applications running on
Intel
(R) 64, Version 11.0    Build 20080930 Package ID: w_cproc_p_11.0.061
Copyright (C) 1985-2008 Intel Corporation.  All rights reserved.
icl: command line warning #10161: unrecognized source type '_configtest.o';
obje
ct file assumed
icl: command line warning #10006: ignoring unknown option
'/LC:\Python25\lib'
icl: command line warning #10006: ignoring unknown option '/LC:\'
icl: command line warning #10006: ignoring unknown option
'/LC:\Python25\libs'

ipo: warning #11009: file format not recognized for _configtest.o
Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:_configtest.exe
_configtest.o
LINK : fatal error LNK1181: cannot open input file '_configtest.o'
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "setup.py", line 96, in <module>
    setup_package()
  File "setup.py", line 89, in setup_package
    configuration=configuration )
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\core.py", line 184, in setup
    return old_setup(**new_attr)
  File "C:\Python25\lib\distutils\core.py", line 151, in setup
    dist.run_commands()
  File "C:\Python25\lib\distutils\dist.py", line 974, in run_commands
    self.run_command(cmd)
  File "C:\Python25\lib\distutils\dist.py", line 994, in run_command
    cmd_obj.run()
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\command\install.py", line 49, in run
    r = old_install.run(self)
  File "C:\Python25\lib\distutils\command\install.py", line 506, in run
    self.run_command('build')
  File "C:\Python25\lib\distutils\cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "C:\Python25\lib\distutils\dist.py", line 994, in run_command
    cmd_obj.run()
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\command\build.py", line 37, in run
    old_build.run(self)
  File "C:\Python25\lib\distutils\command\build.py", line 112, in run
    self.run_command(cmd_name)
  File "C:\Python25\lib\distutils\cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "C:\Python25\lib\distutils\dist.py", line 994, in run_command
    cmd_obj.run()
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\command\build_src.py", line 130, in run
    self.build_sources()
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\command\build_src.py", line 147, in build_sources
    self.build_extension_sources(ext)
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\command\build_src.py", line 250, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "C:\Documents and
Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy
\distutils\command\build_src.py", line 307, in generate_sources
    source = func(extension, build_dir)
  File "numpy\core\setup.py", line 87, in generate_config_h
    raise SystemError,"Failed to test configuration. "\
SystemError: Failed to test configuration. See previous error messages for
more
information.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090127/1258c1c0/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Tue Jan 27 21:39:21 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 28 Jan 2009 11:39:21 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
Message-ID: <497FC559.4000801@ar.media.kyoto-u.ac.jp>

Michael Colonno wrote:
>    Hi ~
>
>    I'm trying to build numpy (hopefully eventually scipy with the same
> setup) with the Intel compilers (and Intel MKL) on the WinXP 64-bit
> platform. Finding / linking to the Intel MKL seems to be successful
> (see below)

Unfortunately, at this stage, it does not say much about linking:
distutils look for files, and do not do any sanity check beyond that.

> but I have an issue with the settings defined somewhere in the various
> setup scripts (can't find where). Per the output below, the Intel
> compilers on Windows are looking for ".obj" object files rather than
> the Linux-style ".o" files.

I think the problem is simply that intel support in numpy for the C
compiler is limited to unix. At least, a quick look at the sources did
not give much informations about windows support: --compiler=intel does
call for the unix version (icc, and this is called first as you can see
in your log). I am actually puzzled: where is the icl.exe coming from ?
grep icl gives nothing in numpy, and nothing in python - did you by any
chance build python itself with the Intel compiler ?

cheers,

David


From mcolonno at gmail.com  Tue Jan 27 22:31:16 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Tue, 27 Jan 2009 19:31:16 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <497FC559.4000801@ar.media.kyoto-u.ac.jp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
Message-ID: <9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>

   Thanks for your response. I manually edited one of the python files
(ccompiler.py I think) to change icc.exe to icl.exe. (This is a trick I used
to use to get F2PY to compile on Windows platforms.) Since icl is a drop-in
replacement for the visual studio compiler / linker, I'd like to edit the
python files configuring this (msvc) but I could not find anything(?) If you
could point me towards the config files(s) for the visual studio compiler
(I'm assuming are configured for the Windows file extensions already) I
could likely make some headway.

   Thanks,
   ~Mike C.

On Tue, Jan 27, 2009 at 6:39 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Michael Colonno wrote:
> >    Hi ~
> >
> >    I'm trying to build numpy (hopefully eventually scipy with the same
> > setup) with the Intel compilers (and Intel MKL) on the WinXP 64-bit
> > platform. Finding / linking to the Intel MKL seems to be successful
> > (see below)
>
> Unfortunately, at this stage, it does not say much about linking:
> distutils look for files, and do not do any sanity check beyond that.
>
> > but I have an issue with the settings defined somewhere in the various
> > setup scripts (can't find where). Per the output below, the Intel
> > compilers on Windows are looking for ".obj" object files rather than
> > the Linux-style ".o" files.
>
> I think the problem is simply that intel support in numpy for the C
> compiler is limited to unix. At least, a quick look at the sources did
> not give much informations about windows support: --compiler=intel does
> call for the unix version (icc, and this is called first as you can see
> in your log). I am actually puzzled: where is the icl.exe coming from ?
> grep icl gives nothing in numpy, and nothing in python - did you by any
> chance build python itself with the Intel compiler ?
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090127/f69cba42/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Jan 28 04:06:00 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 28 Jan 2009 18:06:00 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
Message-ID: <49801FF8.8040805@ar.media.kyoto-u.ac.jp>

Michael Colonno wrote:
>    Thanks for your response. I manually edited one of the python files
> (ccompiler.py I think) to change icc.exe to icl.exe. (This is a trick
> I used to use to get F2PY to compile on Windows platforms.) Since icl
> is a drop-in replacement for the visual studio compiler / linker, I'd
> like to edit the python files configuring this (msvc) but I could not
> find anything(?) If you could point me towards the config files(s) for
> the visual studio compiler (I'm assuming are configured for the
> Windows file extensions already) I could likely make some headway.`

Unfortunately, the code for our building process is difficult to grasp -
there is a lof of magic. Everything is in numpy/distutils. Basically,
you need to create a new compiler, a bit like intelccompiler.py, but for
Windows. I unfortunately can't help you more ATM, since I don't know the
intel compiler on Windows.

cheers,

David


From mcolonno at gmail.com  Wed Jan 28 11:12:54 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Wed, 28 Jan 2009 08:12:54 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <49801FF8.8040805@ar.media.kyoto-u.ac.jp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
Message-ID: <9cca9e840901280812y93736dek1783b5d3b2234de9@mail.gmail.com>

On Wed, Jan 28, 2009 at 1:06 AM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Michael Colonno wrote:
> >    Thanks for your response. I manually edited one of the python files
> > (ccompiler.py I think) to change icc.exe to icl.exe. (This is a trick
> > I used to use to get F2PY to compile on Windows platforms.) Since icl
> > is a drop-in replacement for the visual studio compiler / linker, I'd
> > like to edit the python files configuring this (msvc) but I could not
> > find anything(?) If you could point me towards the config files(s) for
> > the visual studio compiler (I'm assuming are configured for the
> > Windows file extensions already) I could likely make some headway.`
>
> Unfortunately, the code for our building process is difficult to grasp -
> there is a lof of magic. Everything is in numpy/distutils. Basically,
> you need to create a new compiler, a bit like intelccompiler.py, but for
> Windows. I unfortunately can't help you more ATM, since I don't know the
> intel compiler on Windows.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090128/12d1d4ba/attachment.html>

From mcolonno at gmail.com  Wed Jan 28 11:18:47 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Wed, 28 Jan 2009 08:18:47 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <49801FF8.8040805@ar.media.kyoto-u.ac.jp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
Message-ID: <9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>

   I think this is doable; thankfully the Intel compilers on Windows and
Linux are very similar in behavior. The exact same build scripts
*should*work fine provided the file extensions (.o --> .obj) and flags
(-L, etc.)
are modified. In terms of syntax this should be an easy thing to do (it was
with the free-standing F2PY) , but I will need some help navigating through
the magic you refer to. I will put some effort into this and write back if I
hit a roadblock. As an aside: how were the Windows 32-bit installers
created? Is it possible to recreate this process changing the target arch
--> x64?

   Thanks again,
   ~Mike C.

On Wed, Jan 28, 2009 at 1:06 AM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Michael Colonno wrote:
> >    Thanks for your response. I manually edited one of the python files
> > (ccompiler.py I think) to change icc.exe to icl.exe. (This is a trick
> > I used to use to get F2PY to compile on Windows platforms.) Since icl
> > is a drop-in replacement for the visual studio compiler / linker, I'd
> > like to edit the python files configuring this (msvc) but I could not
> > find anything(?) If you could point me towards the config files(s) for
> > the visual studio compiler (I'm assuming are configured for the
> > Windows file extensions already) I could likely make some headway.`
>
> Unfortunately, the code for our building process is difficult to grasp -
> there is a lof of magic. Everything is in numpy/distutils. Basically,
> you need to create a new compiler, a bit like intelccompiler.py, but for
> Windows. I unfortunately can't help you more ATM, since I don't know the
> intel compiler on Windows.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090128/04584d1d/attachment.html>

From timmichelsen at gmx-topmail.de  Wed Jan 28 15:56:04 2009
From: timmichelsen at gmx-topmail.de (Timmie)
Date: Wed, 28 Jan 2009 20:56:04 +0000 (UTC)
Subject: [Numpy-discussion] optimise operation in array with datetime objects
Message-ID: <loom.20090128T205523-747@post.gmane.org>

Hello,
I have an array of datetime objects.

What is the most efficient way of creating a new array
 with only the hours or minutes out of it?

Here is an example:

### imports
import numpy as np
import datetime as dt

### create some data
d = dt.datetime.now()

dates_li = []
count = 0
for i in range(0, 24):
	count = count +1
	fact = count * 3600
	date_new = d + dt.timedelta(0, fact)
	print date_new
	dates_li.append(date_new)

### the array with datetime objects
dates_array = np.array(dates_li)


### this is the loop I would like to optimize:
### looping over arrays is considered inefficient.
### what could be a better way?
hours_array = dates_array.copy()
for i in range(0, dates_array.size):
	hours_array[i] = dates_array[i].hour

hours_array = hours_array.astype('int')

####
	
Thanks in advance for any hints!

Kind regards,
Timmie



From pgmdevlist at gmail.com  Wed Jan 28 16:25:19 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 28 Jan 2009 16:25:19 -0500
Subject: [Numpy-discussion] optimise operation in array with datetime
	objects
In-Reply-To: <loom.20090128T205523-747@post.gmane.org>
References: <loom.20090128T205523-747@post.gmane.org>
Message-ID: <1DE6106D-8096-4F94-A9C1-39979D408039@gmail.com>


On Jan 28, 2009, at 3:56 PM, Timmie wrote:
> ### this is the loop I would like to optimize:
> ### looping over arrays is considered inefficient.
> ### what could be a better way?
> hours_array = dates_array.copy()
> for i in range(0, dates_array.size):
> 	hours_array[i] = dates_array[i].hour

You could try:
np.fromiter((_.hour for _ in dates_li), dtype=np.int)
or
np.array([_.hour for _ in dates_li], dtype=np.int)



From timmichelsen at gmx-topmail.de  Wed Jan 28 17:43:52 2009
From: timmichelsen at gmx-topmail.de (Timmie)
Date: Wed, 28 Jan 2009 22:43:52 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?optimise_operation_in_array_with_dat?=
	=?utf-8?q?etime=09objects?=
References: <loom.20090128T205523-747@post.gmane.org>
	<1DE6106D-8096-4F94-A9C1-39979D408039@gmail.com>
Message-ID: <loom.20090128T223821-184@post.gmane.org>

Thanks!

> > ### this is the loop I would like to optimize:
> > ### looping over arrays is considered inefficient.
> > ### what could be a better way?
> > hours_array = dates_array.copy()
> > for i in range(0, dates_array.size):
> > 	hours_array[i] = dates_array[i].hour
> 
> You could try:
> np.fromiter((_.hour for _ in dates_li), dtype=np.int)
> or
> np.array([_.hour for _ in dates_li], dtype=np.int)

I used dates_li only for the preparation of example data.

So let's suppose I have the array "dates_array" returned from a
a function.

How can the last part be improved:


hours_array = dates_array.copy()
for i in range(0, dates_array.size):
	hours_array[i] = dates_array[i].hour


Or is such a loop accepable from the point of calculation efficiency?

Thanks and greetings,
Timmie





From pgmdevlist at gmail.com  Wed Jan 28 17:49:53 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 28 Jan 2009 17:49:53 -0500
Subject: [Numpy-discussion] optimise operation in array with
	datetime	objects
In-Reply-To: <loom.20090128T223821-184@post.gmane.org>
References: <loom.20090128T205523-747@post.gmane.org>
	<1DE6106D-8096-4F94-A9C1-39979D408039@gmail.com>
	<loom.20090128T223821-184@post.gmane.org>
Message-ID: <BDE13A9B-0AAD-406A-9FB6-9F1BF7914B94@gmail.com>


On Jan 28, 2009, at 5:43 PM, Timmie wrote:
>>
>> You could try:
>> np.fromiter((_.hour for _ in dates_li), dtype=np.int)
>> or
>> np.array([_.hour for _ in dates_li], dtype=np.int)
>
> I used dates_li only for the preparation of example data.
>
> So let's suppose I have the array "dates_array" returned from a
> a function.

Just use dates_array instead of dates_li, then.

>
> hours_array = dates_array.copy()
> for i in range(0, dates_array.size):
> 	hours_array[i] = dates_array[i].hour
>

* What's the point of making a copy of dates_array ? dates_array is a  
ndarray of object, right ? And you want to take the hours, so you  
should have an ndarray of integers for hours_array.
* The issue I have with this part is that you have several calls to  
__getitem__ at each iteration. It might be faster to use create  
hours_array as a block:
hours_array=np.array([_.hour for _ in dates_array], dtype=np.int)




From cournape at gmail.com  Wed Jan 28 19:36:08 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 29 Jan 2009 09:36:08 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
Message-ID: <5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>

On Thu, Jan 29, 2009 at 1:18 AM, Michael Colonno <mcolonno at gmail.com> wrote:
>    I think this is doable; thankfully the Intel compilers on Windows and
> Linux are very similar in behavior.

The problem is that distutils does not abstract this kind of things:
you have a CCompiler class, and a subclass Unix C Compiler, and then
Intel Compiler. OTOH, the MS compiler is its own class which inherit
from CCompiler - all windows specifics are encoded in this class. So I
am afraid you will need to recreate all this class implementation for
Intel C Compiler, because contrary to the Linux case, nothing is
abstracted for windows.

> As an aside: how were the Windows 32-bit installers created?

With mingw compiler.

> Is
> it possible to recreate this process changing the target arch --> x64?

If you can build numpy with the Intel compiler, building the installer
should be a no-brainer.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Wed Jan 28 21:11:02 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 29 Jan 2009 11:11:02 +0900
Subject: [Numpy-discussion] A buildbot farm with shell access - for free ?
Message-ID: <49811036.5030701@ar.media.kyoto-u.ac.jp>

Hi,

    Just saw that on one ML:

http://www.snakebite.org/
http://mail.python.org/pipermail/python-committers/2009-January/000331.html

Bottom line: it looks like there is a set of machines which were donated
to the PSF for buildbot *with shell access* so that people can fix
problems appearing on some platforms. If you look at the email, there
are some 'exotic' machines that mere mortals cannot have access to (like
True64 on Itanium: to quote the email "massive quad Itanium 2 RX-5670s,
chock full of 73GB 15k disks and no less than 78GB of RAM between the
two servers; 32GB in one and 46GB in the other"). There are also windows
machines available.

It is said in the email that this is reserved to the python project, and
prominent python projects like Twisted and Django. Would it be ok to try
to be qualified as a prominent python project as well ?

cheers,

David


From charlesr.harris at gmail.com  Wed Jan 28 21:31:52 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 28 Jan 2009 19:31:52 -0700
Subject: [Numpy-discussion] A buildbot farm with shell access - for free
	?
In-Reply-To: <49811036.5030701@ar.media.kyoto-u.ac.jp>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140901281831j10e35722t9a65165a63259f3a@mail.gmail.com>

On Wed, Jan 28, 2009 at 7:11 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Hi,
>
>    Just saw that on one ML:
>
> http://www.snakebite.org/
> http://mail.python.org/pipermail/python-committers/2009-January/000331.html
>
> Bottom line: it looks like there is a set of machines which were donated
> to the PSF for buildbot *with shell access* so that people can fix
> problems appearing on some platforms. If you look at the email, there
> are some 'exotic' machines that mere mortals cannot have access to (like
> True64 on Itanium: to quote the email "massive quad Itanium 2 RX-5670s,
> chock full of 73GB 15k disks and no less than 78GB of RAM between the
> two servers; 32GB in one and 46GB in the other"). There are also windows
> machines available.
>
> It is said in the email that this is reserved to the python project, and
> prominent python projects like Twisted and Django. Would it be ok to try
> to be qualified as a prominent python project as well ?
>

Ohhh... I love buildbots. Go for it.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090128/45112f0c/attachment.html>

From eads at soe.ucsc.edu  Wed Jan 28 22:20:47 2009
From: eads at soe.ucsc.edu (Damian Eads)
Date: Wed, 28 Jan 2009 19:20:47 -0800
Subject: [Numpy-discussion] A buildbot farm with shell access - for free
	?
In-Reply-To: <49811036.5030701@ar.media.kyoto-u.ac.jp>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
Message-ID: <91b4b1ab0901281920w9d68fd7wa4e58eaa2ee8d52a@mail.gmail.com>

Sounds like a great idea!

On 1/28/09, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>     Just saw that on one ML:
>
> http://www.snakebite.org/
> http://mail.python.org/pipermail/python-committers/2009-January/000331.html
>
> Bottom line: it looks like there is a set of machines which were donated
> to the PSF for buildbot *with shell access* so that people can fix
> problems appearing on some platforms. If you look at the email, there
> are some 'exotic' machines that mere mortals cannot have access to (like
> True64 on Itanium: to quote the email "massive quad Itanium 2 RX-5670s,
> chock full of 73GB 15k disks and no less than 78GB of RAM between the
> two servers; 32GB in one and 46GB in the other"). There are also windows
> machines available.
>
> It is said in the email that this is reserved to the python project, and
> prominent python projects like Twisted and Django. Would it be ok to try
> to be qualified as a prominent python project as well ?
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

-- 
Sent from my mobile device

-----------------------------------------------------
Damian Eads                             Ph.D. Student
Jack Baskin School of Engineering, UCSC        E2-489
1156 High Street                 Machine Learning Lab
Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads


From millman at berkeley.edu  Wed Jan 28 22:55:08 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 28 Jan 2009 19:55:08 -0800
Subject: [Numpy-discussion] A buildbot farm with shell access - for free
	?
In-Reply-To: <49811036.5030701@ar.media.kyoto-u.ac.jp>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
Message-ID: <c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>

On Wed, Jan 28, 2009 at 6:11 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> It is said in the email that this is reserved to the python project, and
> prominent python projects like Twisted and Django. Would it be ok to try
> to be qualified as a prominent python project as well ?

That would be great.


From simpson at math.toronto.edu  Wed Jan 28 23:59:18 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Wed, 28 Jan 2009 23:59:18 -0500
Subject: [Numpy-discussion] convolution axis
Message-ID: <71D5E992-7F18-40DF-9C6F-C3164D07F9F2@math.toronto.edu>

Is there an easy way to perform convolutions along a particular axis  
of an array?

-gideon



From pgmdevlist at gmail.com  Thu Jan 29 00:28:46 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 29 Jan 2009 00:28:46 -0500
Subject: [Numpy-discussion] Documentation: objects.inv ?
Message-ID: <0BFBC3A1-60EA-4E65-8E16-6C881976A129@gmail.com>

All,
Is there an objects.inv lying around for the numpy reference guide, or  
should I start one from scratch ?
Thx a lot in advance
P.



From f.yw at hotmail.com  Thu Jan 29 00:52:42 2009
From: f.yw at hotmail.com (frank wang)
Date: Wed, 28 Jan 2009 22:52:42 -0700
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
Message-ID: <BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>


Hi,
 
I have to buidl a grid with 256 point by the command:
a = arange(-15,16,2)
L = len(a)
cnstl = a.reshape(L,1)+1j*a
 
My problem is that I have a big data array that contains the data round the points in cnstl. I want to slice the point to the closest cnstl point and also compute the error. The condition is in the middle of the two point in x and y axis. I can do it in a for loop. Since Python and numpy have a lot of magic, I want to find an efficient way to do. This problem arise from QAM 256 modulation.
 
Thanks
 
Frank
_________________________________________________________________
Windows Live? Hotmail?:?more than just e-mail. 
http://windowslive.com/explore?ocid=TXT_TAGLM_WL_t2_hm_justgotbetter_explore_012009
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090128/e04745e8/attachment.html>

From robert.kern at gmail.com  Thu Jan 29 00:57:16 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 28 Jan 2009 23:57:16 -0600
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
Message-ID: <3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>

On Wed, Jan 28, 2009 at 23:52, frank wang <f.yw at hotmail.com> wrote:
>
> Hi,
>
> I have to buidl a grid with 256 point by the command:
> a = arange(-15,16,2)
> L = len(a)
> cnstl = a.reshape(L,1)+1j*a
>
> My problem is that I have a big data array that contains the data round the
> points in cnstl. I want to slice the point to the closest cnstl point and
> also compute the error. The condition is in the middle of the two point in x
> and y axis. I can do it in a for loop. Since Python and numpy have a lot of
> magic, I want to find an efficient way to do. This problem arise from QAM
> 256 modulation.

Can you show us the for loop? I'm not really sure what you want to compute.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From f.yw at hotmail.com  Thu Jan 29 01:09:01 2009
From: f.yw at hotmail.com (frank wang)
Date: Wed, 28 Jan 2009 23:09:01 -0700
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl> 
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
Message-ID: <BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>


Here is the for loop that I am think about. Also, I do not know whether the where commands can handle the complicated logic.
The where command basically find the data in the square around the point cnstl[j].
 
Let the data array is qam with size N
 
Out = X
error = X
for i in arange(N):
    for j in arange(L):
         aa = np.where((real(X)<real(cnstl[j])+1) & (real(X)>real(cnstl[j])-1) & (imag(X)<imag(cnstl[j])+1) & (imag(X)>imag(cnstl[j]-1))         Out[aa]=cnstl[j]
         error[aa]=abs(X)**2 - abs(cnstl[j])**2
 
 
Thanks
 
Frank
> Date: Wed, 28 Jan 2009 23:57:16 -0600> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] help on fast slicing on a grid> > On Wed, Jan 28, 2009 at 23:52, frank wang <f.yw at hotmail.com> wrote:> >> > Hi,> >> > I have to buidl a grid with 256 point by the command:> > a = arange(-15,16,2)> > L = len(a)> > cnstl = a.reshape(L,1)+1j*a> >> > My problem is that I have a big data array that contains the data round the> > points in cnstl. I want to slice the point to the closest cnstl point and> > also compute the error. The condition is in the middle of the two point in x> > and y axis. I can do it in a for loop. Since Python and numpy have a lot of> > magic, I want to find an efficient way to do. This problem arise from QAM> > 256 modulation.> > Can you show us the for loop? I'm not really sure what you want to compute.> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion
_________________________________________________________________
Windows Live?: E-mail. Chat. Share. Get more ways to connect. 
http://windowslive.com/howitworks?ocid=TXT_TAGLM_WL_t2_allup_howitworks_012009
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090128/838165ee/attachment.html>

From robert.kern at gmail.com  Thu Jan 29 01:15:48 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 29 Jan 2009 00:15:48 -0600
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>
Message-ID: <3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>

On Thu, Jan 29, 2009 at 00:09, frank wang <f.yw at hotmail.com> wrote:
> Here is the for loop that I am think about. Also, I do not know whether the
> where commands can handle the complicated logic.
> The where command basically find the data in the square around the point
> cnstl[j].

cnstl is a 2D array from your previous description.

> Let the data array is qam with size N

I don't see qam anywhere. Did you mean X?

> Out = X
> error = X

Don't you want something like zeros_like(X) for these?

> for i in arange(N):
>     for j in arange(L):
>          aa = np.where((real(X)<real(cnstl[j])+1) &
> (real(X)>real(cnstl[j])-1) & (imag(X)<imag(cnstl[j])+1) &
> (imag(X)>imag(cnstl[j]-1))
>          Out[aa]=cnstl[j]
>          error[aa]=abs(X)**2 - abs(cnstl[j])**2

I'm still confused. Can you show me a complete, working script with
possibly fake data?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From nadavh at visionsense.com  Thu Jan 29 01:18:33 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu, 29 Jan 2009 08:18:33 +0200
Subject: [Numpy-discussion] convolution axis
References: <71D5E992-7F18-40DF-9C6F-C3164D07F9F2@math.toronto.edu>
Message-ID: <710F2847B0018641891D9A216027636029C3FE@ex3.envision.co.il>

There are at least two options:
1. use convolve1d from numpy.numarray.nd_image (or scipy.ndimage)
2. use scipy.signal.convolve and adjust the dimensions of the convolution kenel to align it along the desired axis.

  Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Gideon Simpson
????: ? 29-?????-09 06:59
??: Discussion of Numerical Python
????: [Numpy-discussion] convolution axis
 
Is there an easy way to perform convolutions along a particular axis  
of an array?

-gideon

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 2888 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090129/9cd4af83/attachment.bin>

From f.yw at hotmail.com  Thu Jan 29 01:28:47 2009
From: f.yw at hotmail.com (frank wang)
Date: Wed, 28 Jan 2009 23:28:47 -0700
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl> 
	<3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
Message-ID: <BLU105-W4063F27AFED247ADEFCB7189C90@phx.gbl>


Hi, Bob,
 
Thanks for your help. 
 
I am sorry for my type error. qam array is the X array in my example.
 
cntl is a complex array contains the point (x,y) axises.
 
I will try to make a workable example. Also I will try to find out the zeros_like function. However, I guess that zeros_like(X) will create an array the same size as X. It it is. Then the two line Out=X and error=X should be Out=zeros_like(X) and error=zeros(X).
 
Also, can where command handel the logic command?
 
aa = np.where((real(X)<real(cnstl[j])+1) & (real(X)>real(cnstl[j])-1) & (imag(X)<imag(cnstl[j])+1) & (imag(X)>imag(cnstl[j]-1))
 
For example, cntl[j]=3+1j*5, then the where command is the same as:
 
aa = np.where((real(X)<4) & (real(X)>2 )& (imag(X)<6) & (imag(X)>4))
 
Thanks
 
Frank> Date: Thu, 29 Jan 2009 00:15:48 -0600> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] help on fast slicing on a grid> > On Thu, Jan 29, 2009 at 00:09, frank wang <f.yw at hotmail.com> wrote:> > Here is the for loop that I am think about. Also, I do not know whether the> > where commands can handle the complicated logic.> > The where command basically find the data in the square around the point> > cnstl[j].> > cnstl is a 2D array from your previous description.> > > Let the data array is qam with size N> > I don't see qam anywhere. Did you mean X?> > > Out = X> > error = X> > Don't you want something like zeros_like(X) for these?> > > for i in arange(N):> > for j in arange(L):> > aa = np.where((real(X)<real(cnstl[j])+1) &> > (real(X)>real(cnstl[j])-1) & (imag(X)<imag(cnstl[j])+1) &> > (imag(X)>imag(cnstl[j]-1))> > Out[aa]=cnstl[j]> > error[aa]=abs(X)**2 - abs(cnstl[j])**2> > I'm still confused. Can you show me a complete, working script with> possibly fake data?> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion
_________________________________________________________________
Windows Live?: E-mail. Chat. Share. Get more ways to connect. 
http://windowslive.com/explore?ocid=TXT_TAGLM_WL_t2_allup_explore_012009
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090128/e3ddd413/attachment.html>

From scott.sinclair.za at gmail.com  Thu Jan 29 01:41:54 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Thu, 29 Jan 2009 08:41:54 +0200
Subject: [Numpy-discussion] make latex in numpy/doc failed
In-Reply-To: <web-116379466@uni-stuttgart.de>
References: <web-116379466@uni-stuttgart.de>
Message-ID: <6a17e9ee0901282241l10e808aauf2b0acbb7fae0596@mail.gmail.com>

> 2009/1/27 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
> a make latex in numpy/doc failed with
>
> ...
>
> Intersphinx hit: PyObject
> http://docs.python.org/dev/c-api/structures.html
> writing... Sphinx error:
> too many nesting section levels for LaTeX, at heading:
> numpy.ma.MaskedArray.__lt__
> make: *** [latex] Fehler 1
>
>
> I am using sphinxv0.5.1
> BTW, make html works fine here.

I see this problem too. It used to work, and I don't think I've
changed anything on my system.

Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.3.0.dev6335'
>>> import sphinx
>>> sphinx.__version__
'0.5.1'

Should I file a ticket, or just let whoever has to build the docs for
the next release sort it out when the time comes?

Cheers,
Scott


From pav at iki.fi  Thu Jan 29 03:17:28 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 29 Jan 2009 08:17:28 +0000 (UTC)
Subject: [Numpy-discussion] Documentation: objects.inv ?
References: <0BFBC3A1-60EA-4E65-8E16-6C881976A129@gmail.com>
Message-ID: <glromo$51v$1@ger.gmane.org>

Thu, 29 Jan 2009 00:28:46 -0500, Pierre GM wrote:
> Is there an objects.inv lying around for the numpy reference guide, or
> should I start one from scratch ?

It's automatically generated by Sphinx, and can be found at

	http://docs.scipy.org/doc/numpy/objects.inv

Let's make the promise that it shall be found there in the future, too.

-- 
Pauli Virtanen



From pgmdevlist at gmail.com  Thu Jan 29 03:58:39 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 29 Jan 2009 03:58:39 -0500
Subject: [Numpy-discussion] Documentation: objects.inv ?
In-Reply-To: <glromo$51v$1@ger.gmane.org>
References: <0BFBC3A1-60EA-4E65-8E16-6C881976A129@gmail.com>
	<glromo$51v$1@ger.gmane.org>
Message-ID: <CFC5BFC5-92B2-4C2D-A20E-0D9CEC2090A3@gmail.com>


On Jan 29, 2009, at 3:17 AM, Pauli Virtanen wrote:

> Thu, 29 Jan 2009 00:28:46 -0500, Pierre GM wrote:
>> Is there an objects.inv lying around for the numpy reference guide,  
>> or
>> should I start one from scratch ?
>
> It's automatically generated by Sphinx, and can be found at
>
> 	http://docs.scipy.org/doc/numpy/objects.inv
>
> Let's make the promise that it shall be found there in the future,  
> too.
>

Got it, thanks a lot.
Pauli, how often is the documentation on docs.scipy.org updated from  
SVN ?
Thx again
P.



From scott.sinclair.za at gmail.com  Thu Jan 29 05:03:48 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Thu, 29 Jan 2009 12:03:48 +0200
Subject: [Numpy-discussion] Documentation: objects.inv ?
In-Reply-To: <CFC5BFC5-92B2-4C2D-A20E-0D9CEC2090A3@gmail.com>
References: <0BFBC3A1-60EA-4E65-8E16-6C881976A129@gmail.com>
	<glromo$51v$1@ger.gmane.org>
	<CFC5BFC5-92B2-4C2D-A20E-0D9CEC2090A3@gmail.com>
Message-ID: <6a17e9ee0901290203i2dbb1638jc9b456de3cfae04a@mail.gmail.com>

> 2009/1/29 Pierre GM <pgmdevlist at gmail.com>:
> Pauli, how often is the documentation on docs.scipy.org updated from
> SVN ?

My understanding is the following:

SVN -> doc-wiki - updated once daily at around 10:00 (UTC?).
doc-wiki -> SVN - infrequently, when someone applies one or more doc
patches produced from the doc-wiki to SVN.

Documentation on docs.scipy.org - infrequently, when someone builds
the docs and posts them.

Cheers,
Scott


From watson.jim at gmail.com  Thu Jan 29 05:11:20 2009
From: watson.jim at gmail.com (James Watson)
Date: Thu, 29 Jan 2009 10:11:20 +0000
Subject: [Numpy-discussion] porting NumPy to Python 3
Message-ID: <f14846790901290211q3bffba43oce25657ed04779dc@mail.gmail.com>

Hi,

I am interested in contributing to the port of NumPy to Python 3.  Who
I should coordinate effort with?

I have started at the Python end of the problem (as opposed to
http://www.scipy.org/Python3k), e.g. I have several patches to get
2to3 to work on NumPy's Python source code.

James.


From william at resolversystems.com  Thu Jan 29 06:43:48 2009
From: william at resolversystems.com (William Reade)
Date: Thu, 29 Jan 2009 11:43:48 +0000
Subject: [Numpy-discussion] Ironclad 0.8 released
Message-ID: <49819674.3040202@resolversystems.com>

Hi all

I'm delighted to announce the release of Ironclad v0.8 -- the 
all-singing, all-dancing CPython API compatibility layer for IronPython 
-- available now from http://code.google.com/p/ironclad/ . Notable 
improvements over the last release include:

* Ironclad is now a neatly self-contained package -- just copy to your 
site-packages and 'import ironclad'.
* No more silly requirement to call ironclad.shutdown() when you're 
finished.
* A few performance improvements.
* Over 900 NumPy tests now pass: in fact, almost all the tests from the 
core, fft, lib, linalg, ma, oldnumeric and random subpackages.
* Over half the .pyds distributed with CPython 2.5 now import cleanly; 
some of them appear to actually work, including _hashlib and _elementtree.

Ironclad grows more stable and mature with every release, and I urge 
IronPython users to try it out and share their impressions: feedback, 
whether positive or negative, is always welcomed.

Cheers
William


From lists at cheimes.de  Thu Jan 29 08:36:07 2009
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 29 Jan 2009 14:36:07 +0100
Subject: [Numpy-discussion] A buildbot farm with shell access - for free
	?
In-Reply-To: <49811036.5030701@ar.media.kyoto-u.ac.jp>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
Message-ID: <glsbc7$4jq$1@ger.gmane.org>

David Cournapeau wrote:
> It is said in the email that this is reserved to the python project, and
> prominent python projects like Twisted and Django. Would it be ok to try
> to be qualified as a prominent python project as well ?

Give it some time. Nobody - not even the Python core devs - have access
to the machines. It's going to take at least several weeks to get the
infrastructure running and to establish a policy.

>From my perspective NumPy and Sage both count as prominent Python
projects. Heck, you are in competition with tools like Matlab and you
ain't looking bad! Furthermore you could make better use of the machines
than Django because you are using way more C extensions and esoteric
libraries.

I recommend you subscribe to the snakebite list and bring up your
interest. You got my +1 already. For now the list is
snakebite-list at googlegroups.com but it will move to another server
(probably Python.org) eventually.

Christian



From mcolonno at gmail.com  Thu Jan 29 10:57:10 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Thu, 29 Jan 2009 07:57:10 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
Message-ID: <9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>

   OK, some progress here. I have two questions: 1) Let's forget about the
Intel compiler(s) for the moment and focus on a standard Windows build.
Python 2.6 comes with two classes in distutils: msvccompiler.py and
msvc9compiler.py. In reading through these, it appears msvc9compiler.py is
ideally suited for what I'm trying to do (use VS 2008 tools). Is it possible
to select this via command line flags with config --compiler=xxx? Choosing
"msvc" *seems* to go for msvccompiler.py (I'm just tyring to trace the magic
as things build). 2) when using the standard msvc setup, things do seem to
work re: setting up the build environemnt (see below). Now, the VS compiler
kicks out a syntax error (output copied below). Any thoughts? This looks
like a peculiarity of the VS compiler but I'm not sure. (I tend to prefer
the Intel C compiler because it is more "Linux-like" and seems to be happier
with cross-platform code.)

   Thanks,
   ~Mike C.

-----------------------

[copying.... output edited for bevity]

running build_ext
No module named msvccompiler in numpy.distutils; trying from distutils
customize MSVCCompiler
customize MSVCCompiler using build_ext
building 'numpy.core.multiarray' extension
compiling C sources
creating build\temp.win-amd64-2.6
creating build\temp.win-amd64-2.6\Release
creating build\temp.win-amd64-2.6\Release\numpy
creating build\temp.win-amd64-2.6\Release\numpy\core
creating build\temp.win-amd64-2.6\Release\numpy\core\src
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
/nolog
o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
-Inumpy\cor
e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-Inumpy\core\src -I
numpy\core\include -IC:\Python26\include -IC:\Python26\PC
/Tcnumpy\core\src\mult
iarraymodule.c
/Fobuild\temp.win-amd64-2.6\Release\numpy\core\src\multiarraymodu
le.obj
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe
/DLL /n
ologo /INCREMENTAL:NO /LIBPATH:C:\Python26\libs
/LIBPATH:C:\Python26\PCbuild\amd
64 /EXPORT:initmultiarray
build\temp.win-amd64-2.6\Release\numpy\core\src\multia
rraymodule.obj /OUT:build\lib.win-amd64-2.6\numpy\core\multiarray.pyd
/IMPLIB:bu
ild\temp.win-amd64-2.6\Release\numpy\core\src\multiarray.lib
/MANIFESTFILE:build
\temp.win-amd64-2.6\Release\numpy\core\src\multiarray.pyd.manifest
mt.exe -nologo -manifest
build\temp.win-amd64-2.6\Release\numpy\core\src\multiar
ray.pyd.manifest
-outputresource:build\lib.win-amd64-2.6\numpy\core\multiarray.p
yd;2
building 'numpy.core.umath' extension
compiling C sources
creating build\temp.win-amd64-2.6\Release\build
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core
creating
build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core\src

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
/nolog
o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
-Inumpy\cor
e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-Inumpy\core\src -I
numpy\core\include -IC:\Python26\include -IC:\Python26\PC
/Tcbuild\src.win-amd64
-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Release\build\src.
win-amd64-2.6\numpy\core\src\umathmodule.obj
umathmodule.c
numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
numpy\core\src\ufuncobject.c(1704) : warning C4244: '=' : conversion from
'npy_i
ntp' to 'int', possible loss of data
numpy\core\src\ufuncobject.c(2425) : warning C4244: '=' : conversion from
'npy_i
ntp' to 'int', possible loss of data
error: Command "C:\Program Files (x86)\Microsoft Visual Studio
9.0\VC\BIN\amd64\
cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG
-Ibuild\src.win-amd64-2.6\numpy\core
\src -Inumpy\core\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-In
umpy\core\src -Inumpy\core\include -IC:\Python26\include -IC:\Python26\PC
/Tcbui
ld\src.win-amd64-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Re
lease\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj" failed with
exit s
tatus 2

-----------------------

On Wed, Jan 28, 2009 at 4:36 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Thu, Jan 29, 2009 at 1:18 AM, Michael Colonno <mcolonno at gmail.com>
> wrote:
> >    I think this is doable; thankfully the Intel compilers on Windows and
> > Linux are very similar in behavior.
>
> The problem is that distutils does not abstract this kind of things:
> you have a CCompiler class, and a subclass Unix C Compiler, and then
> Intel Compiler. OTOH, the MS compiler is its own class which inherit
> from CCompiler - all windows specifics are encoded in this class. So I
> am afraid you will need to recreate all this class implementation for
> Intel C Compiler, because contrary to the Linux case, nothing is
> abstracted for windows.
>
> > As an aside: how were the Windows 32-bit installers created?
>
> With mingw compiler.
>
> > Is
> > it possible to recreate this process changing the target arch --> x64?
>
> If you can build numpy with the Intel compiler, building the installer
> should be a no-brainer.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090129/adef796d/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Thu Jan 29 10:59:28 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 30 Jan 2009 00:59:28 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>	<497FC559.4000801@ar.media.kyoto-u.ac.jp>	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
Message-ID: <4981D260.1040903@ar.media.kyoto-u.ac.jp>

Michael Colonno wrote:
>    OK, some progress here. I have two questions: 1) Let's forget about
> the Intel compiler(s) for the moment and focus on a standard Windows
> build. Python 2.6 comes with two classes in distutils: msvccompiler.py
> and msvc9compiler.py. In reading through these, it appears
> msvc9compiler.py is ideally suited for what I'm trying to do (use VS
> 2008 tools). Is it possible to select this via command line flags with
> config --compiler=xxx?

No - python-distutils normally builds extensions with the same compiler
as the one used to build python itself. Which means VS 2008 for python
2.6, VS 2003 .Net for 2.5 (except for 64 bits which uses a variant of VS
2005). You *cannot* build an extension with VS 2008 for a python built
with VS 2003, for various fundamental reasons.

>  Any thoughts? This looks like a peculiarity of the VS compiler but
> I'm not sure. (I tend to prefer the Intel C compiler because it is
> more "Linux-like" and seems to be happier with cross-platform code.)

Most numpy/scipy developers use gcc compilers on most platforms, so
sometimes some things which do not pass with another compiler slip in.
It is possible that this is the case here - but I could build numpy with
VS 2008 on windows x64 a few weeks ago, so it should only be a
relatively small regression. I will look at it,

David


From simpson at math.toronto.edu  Thu Jan 29 11:22:05 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Thu, 29 Jan 2009 11:22:05 -0500
Subject: [Numpy-discussion] convolution axis
In-Reply-To: <710F2847B0018641891D9A216027636029C3FE@ex3.envision.co.il>
References: <71D5E992-7F18-40DF-9C6F-C3164D07F9F2@math.toronto.edu>
	<710F2847B0018641891D9A216027636029C3FE@ex3.envision.co.il>
Message-ID: <A727F88D-005B-4CD6-BEA5-3D6B97089EAA@math.toronto.edu>

The first option doesn't accept complex data.
-gideon

On Jan 29, 2009, at 1:18 AM, Nadav Horesh wrote:

> There are at least two options:
> 1. use convolve1d from numpy.numarray.nd_image (or scipy.ndimage)
> 2. use scipy.signal.convolve and adjust the dimensions of the  
> convolution kenel to align it along the desired axis.
>
>  Nadav
>
>
> -----????? ??????-----
> ???: numpy-discussion-bounces at scipy.org ??? Gideon Simpson
> ????: ? 29-?????-09 06:59
> ??: Discussion of Numerical Python
> ????: [Numpy-discussion] convolution axis
>
> Is there an easy way to perform convolutions along a particular axis
> of an array?
>
> -gideon
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> <winmail.dat>_______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From mcolonno at gmail.com  Thu Jan 29 11:44:10 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Thu, 29 Jan 2009 08:44:10 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <4981D260.1040903@ar.media.kyoto-u.ac.jp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
Message-ID: <9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>

   OK, I think I have a much better understanding of how this all gets
assembled now. I've got the build environment using both Intel compilers
(C++ and Fortran) and linked to the Intel MKL. Using the Intel C compiler
(icl.exe, more "gcc-like") as a replacement cl.exe (it supports the same
options, flags, and such thankfully), the build proceeds further, but
eventually kicks out with the syntax / parsing error copied below. Let me
know what you think.

   Thanks!
   ~Mike C.

----------------------------------

[edited]

building 'numpy.core.umath' extension
compiling C sources
creating build\temp.win-amd64-2.6\Release\build
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy
creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core
creating
build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core\src

C:\Program Files (x86)\Intel\Compiler\11.0\061\cpp\Bin\intel64\icl.exe /c
/nolog
o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
-Inumpy\cor
e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-Inumpy\core\src -I
numpy\core\include -IC:\Python26\include -IC:\Python26\PC
/Tcbuild\src.win-amd64
-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Release\build\src.
win-amd64-2.6\numpy\core\src\umathmodule.obj
umathmodule.c
numpy\core\src\umathmodule.c.src(64): error: expected an identifier
  static float frexpf(float x, int * i)
               ^

numpy\core\src\umathmodule.c.src(64): error: expected a ")"
  static float frexpf(float x, int * i)
               ^

numpy\core\src\umathmodule.c.src(65): error: expected a ";"
  {
  ^

numpy\core\src\umathmodule.c.src(84): warning #12: parsing restarts here
after p
revious syntax error
  double log1p(double);
                      ^

numpy\core\include\numpy/ufuncobject.h(358): warning #177: function
"generate_ov
erflow_error" was declared but never referenced
  static void generate_overflow_error(void) {
              ^

compilation aborted for build\src.win-amd64-2.6\numpy\core\src\umathmodule.c
(co
de 2)
error: Command "C:\Program Files
(x86)\Intel\Compiler\11.0\061\cpp\Bin\intel64\i
cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG
-Ibuild\src.win-amd64-2.6\numpy\core
\src -Inumpy\core\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-In
umpy\core\src -Inumpy\core\include -IC:\Python26\include -IC:\Python26\PC
/Tcbui
ld\src.win-amd64-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Re
lease\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj" failed with
exit s
tatus 2

----------------------------------

On Thu, Jan 29, 2009 at 7:59 AM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Michael Colonno wrote:
> >    OK, some progress here. I have two questions: 1) Let's forget about
> > the Intel compiler(s) for the moment and focus on a standard Windows
> > build. Python 2.6 comes with two classes in distutils: msvccompiler.py
> > and msvc9compiler.py. In reading through these, it appears
> > msvc9compiler.py is ideally suited for what I'm trying to do (use VS
> > 2008 tools). Is it possible to select this via command line flags with
> > config --compiler=xxx?
>
> No - python-distutils normally builds extensions with the same compiler
> as the one used to build python itself. Which means VS 2008 for python
> 2.6, VS 2003 .Net for 2.5 (except for 64 bits which uses a variant of VS
> 2005). You *cannot* build an extension with VS 2008 for a python built
> with VS 2003, for various fundamental reasons.
>
> >  Any thoughts? This looks like a peculiarity of the VS compiler but
> > I'm not sure. (I tend to prefer the Intel C compiler because it is
> > more "Linux-like" and seems to be happier with cross-platform code.)
>
> Most numpy/scipy developers use gcc compilers on most platforms, so
> sometimes some things which do not pass with another compiler slip in.
> It is possible that this is the case here - but I could build numpy with
> VS 2008 on windows x64 a few weeks ago, so it should only be a
> relatively small regression. I will look at it,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090129/ce2f3e57/attachment.html>

From matthieu.brucher at gmail.com  Thu Jan 29 11:50:21 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 29 Jan 2009 17:50:21 +0100
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
	<9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
Message-ID: <e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>

> numpy\core\src\umathmodule.c.src(64): error: expected an identifier
>   static float frexpf(float x, int * i)
>                ^
>
> numpy\core\src\umathmodule.c.src(64): error: expected a ")"
>   static float frexpf(float x, int * i)
>                ^
>
> numpy\core\src\umathmodule.c.src(65): error: expected a ";"
>   {
>   ^

Perhaps a macro that is replaced? Visual Studio defines min, max as
macros. Maybe it is the same for frexpf? Could you check in the
preprocessed C++ file ? /E to generate it Intel Compiler, I think.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From mcolonno at gmail.com  Thu Jan 29 12:43:35 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Thu, 29 Jan 2009 09:43:35 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
	<9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
	<e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>
Message-ID: <9cca9e840901290943i3b34863bu80125745e4ac9716@mail.gmail.com>

   Not sure if it's a VS thing since I get different compile errors with
either the MS or Intel C compiler under the same environment (Visual Studio
for x64 console). Let me start with a more basic question: I'm using the
package available on SourceForge (1.2.1, 2008-10-29 14:36). If it can be
confirmed this package builds happily via VS 2008 x64 console, can I get a
look at the build log and/or configuration? I should be able to duplicate
this with identical tools (I hope). If it makes a difference I will try to
grab the latest & greatest source instead.

   Thanks,
   ~Mike C.


On Thu, Jan 29, 2009 at 8:50 AM, Matthieu Brucher <
matthieu.brucher at gmail.com> wrote:

> > numpy\core\src\umathmodule.c.src(64): error: expected an identifier
> >   static float frexpf(float x, int * i)
> >                ^
> >
> > numpy\core\src\umathmodule.c.src(64): error: expected a ")"
> >   static float frexpf(float x, int * i)
> >                ^
> >
> > numpy\core\src\umathmodule.c.src(65): error: expected a ";"
> >   {
> >   ^
>
> Perhaps a macro that is replaced? Visual Studio defines min, max as
> macros. Maybe it is the same for frexpf? Could you check in the
> preprocessed C++ file ? /E to generate it Intel Compiler, I think.
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090129/aef0a591/attachment.html>

From cournape at gmail.com  Thu Jan 29 13:05:36 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 30 Jan 2009 03:05:36 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901290943i3b34863bu80125745e4ac9716@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
	<9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
	<e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>
	<9cca9e840901290943i3b34863bu80125745e4ac9716@mail.gmail.com>
Message-ID: <5b8d13220901291005s4907a6dag3ae3fa1bba799ca0@mail.gmail.com>

On Fri, Jan 30, 2009 at 2:43 AM, Michael Colonno <mcolonno at gmail.com> wrote:
>    Not sure if it's a VS thing since I get different compile errors with
> either the MS or Intel C compiler under the same environment (Visual Studio
> for x64 console). Let me start with a more basic question: I'm using the
> package available on SourceForge (1.2.1, 2008-10-29 14:36). If it can be
> confirmed this package builds happily via VS 2008 x64 console, can I get a
> look at the build log and/or configuration? I should be able to duplicate
> this with identical tools (I hope). If it makes a difference I will try to
> grab the latest & greatest source instead.

Oh, you should forget about 1.2.1 for windows x64.  A lot of fixes
have been applied since then, and in any case, I won't try to fix
1.2.1 build problems. You should definitely use svn if you care about
windows x64 - I hope 1.3.0 will be fully supported on that platform.

David


From timmichelsen at gmx-topmail.de  Thu Jan 29 18:46:31 2009
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Fri, 30 Jan 2009 00:46:31 +0100
Subject: [Numpy-discussion] Ironclad 0.8 released
In-Reply-To: <49819674.3040202@resolversystems.com>
References: <49819674.3040202@resolversystems.com>
Message-ID: <49823FD7.3050101@gmx-topmail.de>

Hello William,

> * A few performance improvements.
> * Over 900 NumPy tests now pass: in fact, almost all the tests from the 
> core, fft, lib, linalg, ma, oldnumeric and random subpackages.
I have some questions here for the science interested. I hope that they 
are not too specific:

* Can you also use scipy with IronClad/ResolverOne (RO)?
* Can you recognize arrays with datetime objects in RO?
* How is datetime information handled in RO?
* What about rpy?

If all the above were working in RO, it would be terrific!

> Ironclad grows more stable and mature with every release, and I urge 
> IronPython users to try it out and share their impressions: feedback, 
> whether positive or negative, is always welcomed.
You may find some information here:
http://groups.google.com/group/python-excel/msg/eb349c229696c7f7

I would also recommend this nice & straightforward video
http://www.resolversystems.com/screencasts/numpy/

Kind regards,
Timmie



From mcolonno at gmail.com  Thu Jan 29 18:54:35 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Thu, 29 Jan 2009 15:54:35 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <5b8d13220901291005s4907a6dag3ae3fa1bba799ca0@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
	<9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
	<e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>
	<9cca9e840901290943i3b34863bu80125745e4ac9716@mail.gmail.com>
	<5b8d13220901291005s4907a6dag3ae3fa1bba799ca0@mail.gmail.com>
Message-ID: <9cca9e840901291554t3e480605ye379f38e1c0c68db@mail.gmail.com>

   OK, I'm on the cusp of success now. I updated to the most recent code and
get to a sequence of errors surround math functions. This must be due to
linking the MKL libs. I tried various combinations (including those
recommended by the MKL docs) but could not get around the errors. Looks like
a missing header file(?) In any event, I hope this can be fixed by getting
the right into in site.cfg. Currently I have:

[mkl]
include_dirs = C:\Program Files
(x86)\Intel\Compiler\11.0\061\cpp\mkl\include
library_dirs = C:\Program Files
(x86)\Intel\Compiler\11.0\061\cpp\mkl\em64t\lib
mkl_libs = mkl_em64t, mkl_dll, mkl_core    # various combinations of other
tried...
lapack_libs = mkl_lapack

   Below is the relevant portion of the output. During config the libraries
check in as "FOUND" so I'm pretty sure the paths above are good.

   Thanks,
   ~Mike C.

---------------------------

[edited]

numpy\core\src\umath_funcs.inc.src(557): warning #266: function "sinhl"
declared
 implicitly
      shr = sinhl(xr);
            ^

numpy\core\src\umath_funcs.inc.src(558): warning #266: function "coshl"
declared
 implicitly
      chr = coshl(xr);
            ^

numpy\core\src\umath_loops.inc.src(913): warning #266: function "floorl"
declare
d implicitly
          *((longdouble *)op1) = floorl(in1/in2);
                                 ^

numpy\core\src\umath_loops.inc.src(923): warning #266: function "fmodl"
declared
 implicitly
          const longdouble res = fmodl(in1,in2);
                                 ^

numpy\core\src\umath_loops.inc.src(1003): warning #266: function "modfl"
declare
d implicitly
          *((longdouble *)op1) = modfl(in1, (longdouble *)op2);
                                 ^

numpy\core\src\umath_loops.inc.src(1207): warning #266: function "fabsf"
declare
d implicitly
          if (fabsf(in1i) <= fabsf(in1r)) {
              ^

numpy\core\src\umath_loops.inc.src(1110): warning #266: function "floorl"
declar
ed implicitly
          ((longdouble *)op1)[0] = floorl((in1r*in2r + in1i*in2i)/d);
                                   ^

numpy\core\src\umath_loops.inc.src(1207): warning #266: function "fabsl"
declare
d implicitly
          if (fabsl(in1i) <= fabsl(in1r)) {
              ^

numpy\core\src\umath_loops.inc.src(1246): warning #266: function "sqrtl"
declare
d implicitly
          *((longdouble *)op1) = sqrtl(in1r*in1r + in1i*in1i);
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(244):
error
: identifier "acosl" is undefined
      arccos_data[2] = (void *) acosl;
                                ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(253):
error
: identifier "acoshf" is undefined
      arccosh_data[0] = (void *) acoshf;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(257):
error
: identifier "acoshl" is undefined
      arccosh_data[2] = (void *) acoshl;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(270):
error
: identifier "asinl" is undefined
      arcsin_data[2] = (void *) asinl;
                                ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(279):
error
: identifier "asinhf" is undefined
      arcsinh_data[0] = (void *) asinhf;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(283):
error
: identifier "asinhl" is undefined
      arcsinh_data[2] = (void *) asinhl;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(296):
error
: identifier "atanl" is undefined
      arctan_data[2] = (void *) atanl;
                                ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(309):
error
: identifier "atan2l" is undefined
      arctan2_data[2] = (void *) atan2l;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(312):
error
: identifier "atanhf" is undefined
      arctanh_data[0] = (void *) atanhf;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(316):
error
: identifier "atanhl" is undefined
      arctanh_data[2] = (void *) atanhl;
                                 ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(335):
error
: identifier "ceill" is undefined
      ceil_data[2] = (void *) ceill;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(343):
error
: identifier "cosl" is undefined
      cos_data[2] = (void *) cosl;
                             ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(356):
error
: identifier "coshl" is undefined
      cosh_data[2] = (void *) coshl;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(385):
error
: identifier "expl" is undefined
      exp_data[2] = (void *) expl;
                             ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(394):
error
: identifier "exp2f" is undefined
      exp2_data[0] = (void *) exp2f;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(396):
error
: identifier "exp2" is undefined
      exp2_data[1] = (void *) exp2;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(398):
error
: identifier "exp2l" is undefined
      exp2_data[2] = (void *) exp2l;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(401):
error
: identifier "expm1f" is undefined
      expm1_data[0] = (void *) expm1f;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(405):
error
: identifier "expm1l" is undefined
      expm1_data[2] = (void *) expm1l;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(414):
error
: identifier "fabsf" is undefined
      fabs_data[0] = (void *) fabsf;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(418):
error
: identifier "fabsl" is undefined
      fabs_data[2] = (void *) fabsl;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(425):
error
: identifier "floorl" is undefined
      floor_data[2] = (void *) floorl;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(438):
error
: identifier "fmodl" is undefined
      fmod_data[12] = (void *) fmodl;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(441):
error
: identifier "hypotf" is undefined
      hypot_data[0] = (void *) hypotf;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(445):
error
: identifier "hypotl" is undefined
      hypot_data[2] = (void *) hypotl;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(456):
error
: identifier "logl" is undefined
      log_data[2] = (void *) logl;
                             ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(469):
error
: identifier "log10l" is undefined
      log10_data[2] = (void *) log10l;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(478):
error
: identifier "log1pf" is undefined
      log1p_data[0] = (void *) log1pf;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(482):
error
: identifier "log1pl" is undefined
      log1p_data[2] = (void *) log1pl;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(491):
error
: identifier "log2f" is undefined
      log2_data[0] = (void *) log2f;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(493):
error
: identifier "log2" is undefined
      log2_data[1] = (void *) log2;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(495):
error
: identifier "log2l" is undefined
      log2_data[2] = (void *) log2l;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(534):
error
: identifier "powl" is undefined
      power_data[12] = (void *) powl;
                                ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(564):
error
: identifier "rintf" is undefined
      rint_data[0] = (void *) rintf;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(566):
error
: identifier "rint" is undefined
      rint_data[1] = (void *) rint;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(568):
error
: identifier "rintl" is undefined
      rint_data[2] = (void *) rintl;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(581):
error
: identifier "sinl" is undefined
      sin_data[2] = (void *) sinl;
                             ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(594):
error
: identifier "sinhl" is undefined
      sinh_data[2] = (void *) sinhl;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(607):
error
: identifier "sqrtl" is undefined
      sqrt_data[2] = (void *) sqrtl;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(624):
error
: identifier "tanl" is undefined
      tan_data[2] = (void *) tanl;
                             ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(637):
error
: identifier "tanhl" is undefined
      tanh_data[2] = (void *) tanhl;
                              ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(648):
error
: identifier "truncf" is undefined
      trunc_data[0] = (void *) truncf;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(650):
error
: identifier "trunc" is undefined
      trunc_data[1] = (void *) trunc;
                               ^

build\src.win-amd64-2.6\numpy\core\include/numpy/__umath_generated.c(652):
error
: identifier "truncl" is undefined
      trunc_data[2] = (void *) truncl;
                               ^

numpy\core\include\numpy/ufuncobject.h(372): warning #177: function
"generate_ov
erflow_error" was declared but never referenced
  static void generate_overflow_error(void) {
              ^

numpy\core\src\umath_funcs.inc.src(70): warning #177: function "exp2_1mf"
was de
clared but never referenced
  exp2_1mf(float x)
  ^

numpy\core\src\umath_funcs.inc.src(70): warning #177: function "exp2_1m" was
dec
lared but never referenced
  exp2_1m(double x)
  ^

numpy\core\src\umath_funcs.inc.src(70): warning #177: function "exp2_1ml"
was de
clared but never referenced
  exp2_1ml(longdouble x)
  ^

compilation aborted for build\src.win-amd64-2.6\numpy\core\src\umathmodule.c
(co
de 2)
error: Command "C:\Program Files
(x86)\Intel\Compiler\11.0\061\cpp\Bin\intel64\i
cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG
-Ibuild\src.win-amd64-2.6\numpy\core
\src -Inumpy\core\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-In
umpy\core\src -Inumpy\core\include -IC:\Python26\include -IC:\Python26\PC
/Tcbui
ld\src.win-amd64-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Re
lease\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj" failed with
exit s
tatus 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090129/5ea3d20a/attachment.html>

From timmichelsen at gmx-topmail.de  Thu Jan 29 19:04:00 2009
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Fri, 30 Jan 2009 01:04:00 +0100
Subject: [Numpy-discussion] Ironclad 0.8 released
In-Reply-To: <49819674.3040202@resolversystems.com>
References: <49819674.3040202@resolversystems.com>
Message-ID: <gltg5h$eie$1@ger.gmane.org>

Hello William,
once again.

I just noticed that Resolver One can only import data from Excel.

In science, the common low level data exchange format is ASCII text 
like: CSV or tab separated.
You may consider adding this.

I did not find any installation instructions contained in the docs for 
the IronClad binaries.
Where am I to put the files?
Can Resolver One also recognize the python installation which is already 
installed on the system?

Thanks in advance for answering!

Kind regards,
Timmie



From mcolonno at gmail.com  Thu Jan 29 19:41:41 2009
From: mcolonno at gmail.com (Michael Colonno)
Date: Thu, 29 Jan 2009 16:41:41 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901291554t3e480605ye379f38e1c0c68db@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
	<9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
	<e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>
	<9cca9e840901290943i3b34863bu80125745e4ac9716@mail.gmail.com>
	<5b8d13220901291005s4907a6dag3ae3fa1bba799ca0@mail.gmail.com>
	<9cca9e840901291554t3e480605ye379f38e1c0c68db@mail.gmail.com>
Message-ID: <9cca9e840901291641n1729b64dm59f6bb5b4896f44b@mail.gmail.com>

   This is the first failure in the build log:

_configtest.c
_configtest.c(1): catastrophic error: could not open source file
"inttypes.h"
  #include <inttypes.h>
                       ^

compilation aborted for _configtest.c (code 4)
failure.

   I don't have this file anywhere on my system; it appears to be a gcc
header file. Any suggestions?

   Thanks,
   ~Mike C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090129/4f401087/attachment.html>

From william at resolversystems.com  Thu Jan 29 22:42:29 2009
From: william at resolversystems.com (William Reade)
Date: Fri, 30 Jan 2009 03:42:29 +0000
Subject: [Numpy-discussion] Ironclad 0.8 released
In-Reply-To: <49823FD7.3050101@gmx-topmail.de>
References: <49819674.3040202@resolversystems.com>
	<49823FD7.3050101@gmx-topmail.de>
Message-ID: <0004ED08-A5AE-4F74-B59A-BF71193F41C1@resolversystems.com>

Hi Tim

Thank you for your interest! To answer your various questions order  
of approximately increasing complexity:

Most importantly: the currently available version of Resolver One  
does not have numpy support built in, and for various reasons I  
wouldn't advise trying to integrate Ironclad with v1.3; please wait  
for 1.4 :-).

I just downloaded R and rpy2; they don't work with Ironclad at the  
moment, and I'm not yet sure why.

Ironclad shouldn't require any special installation: just copy the  
ironclad package (containing __init__.py, python25.dll and  
ironclad.dll) to wherever you normally put packages for IronPython (I  
personally have my IRONPYTHONPATH pointing to the appropriate  
subdirectories of a standard Python 2.5 install). Clearly my  
documentation is currently inadequate; I'll have to work on that for  
the next release.

RO uses .NET DateTime objects by default, but it's perfectly possible  
to use python datetime objects instead; I can confirm that you can  
put both those datatypes into numpy arrays.

While RO doesn't have an explicit CSV/TSV import option, you should  
be able to paste data in those formats directly into your worksheets;  
and, as soon as we have numpy integration out in the wild, you should  
be able to create ndarrays directly from your data files.

Scipy support is coming along surprisingly nicely; I'd sort of  
forgotten about it until a couple of people asked me about it earlier  
today, and it turns out -- to my surprise and delight -- that it took  
very little work to get the SVN head running over 800 of the scipy  
tests (even if you can't just 'import scipy' without errors). Sadly,  
a lot of the modules seem to use scipy.sparse, which I can't  
import... yet ;-).

So, we haven't built any scipy support into Resolver One yet, but  
hopefully we'll be able to do so without too much pain.

Finally, I'm glad you like my screencast :-). I hope I haven't missed  
anything; let me know if I have, or if I can help with anything else.

William

On 29 Jan 2009, at 23:46, Tim Michelsen wrote:

> Hello William,
>
>> * A few performance improvements.
>> * Over 900 NumPy tests now pass: in fact, almost all the tests  
>> from the core, fft, lib, linalg, ma, oldnumeric and random  
>> subpackages.
> I have some questions here for the science interested. I hope that  
> they are not too specific:
>
> * Can you also use scipy with IronClad/ResolverOne (RO)?
> * Can you recognize arrays with datetime objects in RO?
> * How is datetime information handled in RO?
> * What about rpy?
>
> If all the above were working in RO, it would be terrific!
>
>> Ironclad grows more stable and mature with every release, and I  
>> urge IronPython users to try it out and share their impressions:  
>> feedback, whether positive or negative, is always welcomed.
> You may find some information here:
> http://groups.google.com/group/python-excel/msg/eb349c229696c7f7
>
> I would also recommend this nice & straightforward video
> http://www.resolversystems.com/screencasts/numpy/
>
> Kind regards,
> Timmie



From cournape at gmail.com  Fri Jan 30 02:20:31 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 30 Jan 2009 16:20:31 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901291641n1729b64dm59f6bb5b4896f44b@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<4981D260.1040903@ar.media.kyoto-u.ac.jp>
	<9cca9e840901290844g1a0f43e9r772f869c2c785d7b@mail.gmail.com>
	<e76aa17f0901290850u3230d082j76a643da57f246b3@mail.gmail.com>
	<9cca9e840901290943i3b34863bu80125745e4ac9716@mail.gmail.com>
	<5b8d13220901291005s4907a6dag3ae3fa1bba799ca0@mail.gmail.com>
	<9cca9e840901291554t3e480605ye379f38e1c0c68db@mail.gmail.com>
	<9cca9e840901291641n1729b64dm59f6bb5b4896f44b@mail.gmail.com>
Message-ID: <5b8d13220901292320t74f796ceg919d224aa87d1ed@mail.gmail.com>

On Fri, Jan 30, 2009 at 9:41 AM, Michael Colonno <mcolonno at gmail.com> wrote:
>    This is the first failure in the build log:
>
> _configtest.c
> _configtest.c(1): catastrophic error: could not open source file
> "inttypes.h"
>   #include <inttypes.h>
>                        ^
>
> compilation aborted for _configtest.c (code 4)
> failure.

That's strange, because configtest.c means it is a configuration test,
and failure just means the tested feature is not available on the
platform - it should certainly not stop the build process. Is this
with Intel compiler ? I think I remembered having seen some bug either
in python or Intel Compiler in the error return code when launching
intel compiler in a subprocess.

I just checked the last svn version on windows 64: it builds OK with
compiler version 15 (either VS 2008 for 64 bits, which I don't have
access to, or with the Platform SDK 6.1, which is free).

David


From hanni.ali at gmail.com  Fri Jan 30 07:32:48 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Fri, 30 Jan 2009 12:32:48 +0000
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
Message-ID: <789d27b10901300432q28226381x829575e8b1fa23fe@mail.gmail.com>

I have been meaning to chip in but so far hadn't got to it so hear goes.

In response to this particular issue I currently use numpy (1.2.1) built
with msvc VS 2008 by simply commenting out these definitions in the
numpy\core\src\umathmodule.c.src

That works just fine and allows me to use the built in lapack light that
comes with numpy on 64-bit windows no problem.

I have spent many hours working on compiling a different lapack/blas
implementation for windows with numpy so far with no joy, so would be very
pleased if we can finally figure this out.

I have previously posted this link on the list:

http://icl.cs.utk.edu/lapack-for-windows/index.html

Using this package the intel visual fortran compiler and msvc C compiler I
have managed to get most of numpy to compile against lapack/blas, but the
process still trips up at linking with the folowwing message:

warning: build_ext: extension 'numpy.linalg.lapack_lite' has Fortran
libraries but no Fortran linker found, using default linker

It complains about missing external symbols.

   Creating library build\temp.win-amd64-2.6\Release\numpy\linalg\
lapack_lite.li
b and object build\temp.win-amd64-2.6\Release\numpy\linalg\lapack_lite.exp
lapack_litemodule.obj : error LNK2019: unresolved external symbol dgeev_
referen
ced in function lapack_lite_dgeev
lapack_litemodule.obj : error LNK2019: unresolved external symbol dsyevd_
refere
nced in function lapack_lite_dsyevd
lapack_litemodule.obj : error LNK2019: unresolved external symbol zheevd_
refere
nced in function lapack_lite_zheevd
lapack_litemodule.obj : error LNK2019: unresolved external symbol dgelsd_
refere
nced in function lapack_lite_dgelsd
lapack_litemodule.obj : error LNK2019: unresolved external symbol dgesv_
referen
ced in function lapack_lite_dgesv
lapack_litemodule.obj : error LNK2019: unresolved external symbol dgesdd_
refere
nced in function lapack_lite_dgesdd
lapack_litemodule.obj : error LNK2019: unresolved external symbol dgetrf_
refere
nced in function lapack_lite_dgetrf
lapack_litemodule.obj : error LNK2019: unresolved external symbol dpotrf_
refere
nced in function lapack_lite_dpotrf
lapack_litemodule.obj : error LNK2019: unresolved external symbol dgeqrf_
refere
nced in function lapack_lite_dgeqrf
lapack_litemodule.obj : error LNK2019: unresolved external symbol dorgqr_
refere
nced in function lapack_lite_dorgqr
lapack_litemodule.obj : error LNK2019: unresolved external symbol zgeev_
referen
ced in function lapack_lite_zgeev
lapack_litemodule.obj : error LNK2019: unresolved external symbol zgelsd_
refere
nced in function lapack_lite_zgelsd
lapack_litemodule.obj : error LNK2019: unresolved external symbol zgesv_
referen
ced in function lapack_lite_zgesv
lapack_litemodule.obj : error LNK2019: unresolved external symbol zgesdd_
refere
nced in function lapack_lite_zgesdd
lapack_litemodule.obj : error LNK2019: unresolved external symbol zgetrf_
refere
nced in function lapack_lite_zgetrf
lapack_litemodule.obj : error LNK2019: unresolved external symbol zpotrf_
refere
nced in function lapack_lite_zpotrf
lapack_litemodule.obj : error LNK2019: unresolved external symbol zgeqrf_
refere
nced in function lapack_lite_zgeqrf
lapack_litemodule.obj : error LNK2019: unresolved external symbol zungqr_
refere
nced in function lapack_lite_zungqr
build\lib.win-amd64-2.6\numpy\linalg\lapack_lite.pyd : fatal error LNK1120:
18 u
nresolved externals
error: Command "C:\Program Files (x86)\Microsoft Visual Studio
9.0\VC\BIN\amd64\
link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:"C:\Program Files
(x86)\Universit
y Of Tennessee\LAPACK 3.1.1\lib\x64" /LIBPATH:C:\Python26\libs
/LIBPATH:C:\Pytho
n26\PCbuild\amd64 LAPACK.lib BLAS.lib /EXPORT:initlapack_lite
build\temp.win-amd
64-2.6\Release\numpy\linalg\lapack_litemodule.obj
/OUT:build\lib.win-amd64-2.6\n
umpy\linalg\lapack_lite.pyd
/IMPLIB:build\temp.win-amd64-2.6\Release\numpy\linal
g\lapack_lite.lib
/MANIFESTFILE:build\temp.win-amd64-2.6\Release\numpy\linalg\la
pack_lite.pyd.manifest" failed with exit status 1120

I suspect persuading setup.py to use the appropriate linker will sort this
out, but I haven't had time to address what - I hope - could be the final
hurdle.

Hanni


2009/1/29 Michael Colonno <mcolonno at gmail.com>

>    OK, some progress here. I have two questions: 1) Let's forget about the
> Intel compiler(s) for the moment and focus on a standard Windows build.
> Python 2.6 comes with two classes in distutils: msvccompiler.py and
> msvc9compiler.py. In reading through these, it appears msvc9compiler.py is
> ideally suited for what I'm trying to do (use VS 2008 tools). Is it possible
> to select this via command line flags with config --compiler=xxx? Choosing
> "msvc" *seems* to go for msvccompiler.py (I'm just tyring to trace the
> magic as things build). 2) when using the standard msvc setup, things do
> seem to work re: setting up the build environemnt (see below). Now, the VS
> compiler kicks out a syntax error (output copied below). Any thoughts? This
> looks like a peculiarity of the VS compiler but I'm not sure. (I tend to
> prefer the Intel C compiler because it is more "Linux-like" and seems to be
> happier with cross-platform code.)
>
>    Thanks,
>    ~Mike C.
>
> -----------------------
>
> [copying.... output edited for bevity]
>
> running build_ext
> No module named msvccompiler in numpy.distutils; trying from distutils
> customize MSVCCompiler
> customize MSVCCompiler using build_ext
> building 'numpy.core.multiarray' extension
> compiling C sources
> creating build\temp.win-amd64-2.6
> creating build\temp.win-amd64-2.6\Release
> creating build\temp.win-amd64-2.6\Release\numpy
> creating build\temp.win-amd64-2.6\Release\numpy\core
> creating build\temp.win-amd64-2.6\Release\numpy\core\src
> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
> /nolog
> o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
> -Inumpy\cor
> e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
> -Inumpy\core\src -I
> numpy\core\include -IC:\Python26\include -IC:\Python26\PC
> /Tcnumpy\core\src\mult
> iarraymodule.c
> /Fobuild\temp.win-amd64-2.6\Release\numpy\core\src\multiarraymodu
> le.obj
> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe
> /DLL /n
> ologo /INCREMENTAL:NO /LIBPATH:C:\Python26\libs
> /LIBPATH:C:\Python26\PCbuild\amd
> 64 /EXPORT:initmultiarray
> build\temp.win-amd64-2.6\Release\numpy\core\src\multia
> rraymodule.obj /OUT:build\lib.win-amd64-2.6\numpy\core\multiarray.pyd
> /IMPLIB:bu
> ild\temp.win-amd64-2.6\Release\numpy\core\src\multiarray.lib
> /MANIFESTFILE:build
> \temp.win-amd64-2.6\Release\numpy\core\src\multiarray.pyd.manifest
> mt.exe -nologo -manifest
> build\temp.win-amd64-2.6\Release\numpy\core\src\multiar
> ray.pyd.manifest
> -outputresource:build\lib.win-amd64-2.6\numpy\core\multiarray.p
> yd;2
> building 'numpy.core.umath' extension
> compiling C sources
> creating build\temp.win-amd64-2.6\Release\build
> creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6
> creating build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy
> creating
> build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core
> creating
> build\temp.win-amd64-2.6\Release\build\src.win-amd64-2.6\numpy\core\src
>
> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
> /nolog
> o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
> -Inumpy\cor
> e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
> -Inumpy\core\src -I
> numpy\core\include -IC:\Python26\include -IC:\Python26\PC
> /Tcbuild\src.win-amd64
> -2.6\numpy\core\src\umathmodule.c
> /Fobuild\temp.win-amd64-2.6\Release\build\src.
> win-amd64-2.6\numpy\core\src\umathmodule.obj
> umathmodule.c
> numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
> numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
> numpy\core\src\ufuncobject.c(1704) : warning C4244: '=' : conversion from
> 'npy_i
> ntp' to 'int', possible loss of data
> numpy\core\src\ufuncobject.c(2425) : warning C4244: '=' : conversion from
> 'npy_i
> ntp' to 'int', possible loss of data
> error: Command "C:\Program Files (x86)\Microsoft Visual Studio
> 9.0\VC\BIN\amd64\
> cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG
> -Ibuild\src.win-amd64-2.6\numpy\core
> \src -Inumpy\core\include
> -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy -In
> umpy\core\src -Inumpy\core\include -IC:\Python26\include -IC:\Python26\PC
> /Tcbui
> ld\src.win-amd64-2.6\numpy\core\src\umathmodule.c
> /Fobuild\temp.win-amd64-2.6\Re
> lease\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj" failed with
> exit s
> tatus 2
>
> -----------------------
>
>
> On Wed, Jan 28, 2009 at 4:36 PM, David Cournapeau <cournape at gmail.com>wrote:
>
>> On Thu, Jan 29, 2009 at 1:18 AM, Michael Colonno <mcolonno at gmail.com>
>> wrote:
>> >    I think this is doable; thankfully the Intel compilers on Windows and
>> > Linux are very similar in behavior.
>>
>> The problem is that distutils does not abstract this kind of things:
>> you have a CCompiler class, and a subclass Unix C Compiler, and then
>> Intel Compiler. OTOH, the MS compiler is its own class which inherit
>> from CCompiler - all windows specifics are encoded in this class. So I
>> am afraid you will need to recreate all this class implementation for
>> Intel C Compiler, because contrary to the Linux case, nothing is
>> abstracted for windows.
>>
>> > As an aside: how were the Windows 32-bit installers created?
>>
>> With mingw compiler.
>>
>> > Is
>> > it possible to recreate this process changing the target arch --> x64?
>>
>> If you can build numpy with the Intel compiler, building the installer
>> should be a no-brainer.
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/ae19bcd7/attachment.html>

From ndbecker2 at gmail.com  Fri Jan 30 08:18:36 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 30 Jan 2009 08:18:36 -0500
Subject: [Numpy-discussion] minor improvment to ones
Message-ID: <gluunc$6t5$1@ger.gmane.org>

A nit, but it would be nice if 'ones' could fill with a value other than 1.

Maybe an optional val= keyword?




From david at ar.media.kyoto-u.ac.jp  Fri Jan 30 08:05:42 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 30 Jan 2009 22:05:42 +0900
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <gluunc$6t5$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>
Message-ID: <4982FB26.1010509@ar.media.kyoto-u.ac.jp>

Neal Becker wrote:
> A nit, but it would be nice if 'ones' could fill with a value other than 1.
>
> Maybe an optional val= keyword?
>   

What would be the advantage compared to fill ? I would guess ones and
zeros are special because those two values are special (they can be
defined for many types, as  neutral elements for + and *),

David



From scott.sinclair.za at gmail.com  Fri Jan 30 08:54:52 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Fri, 30 Jan 2009 15:54:52 +0200
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <4982FB26.1010509@ar.media.kyoto-u.ac.jp>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
Message-ID: <6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>

> 2009/1/30 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> Neal Becker wrote:
>> A nit, but it would be nice if 'ones' could fill with a value other than 1.
>>
>> Maybe an optional val= keyword?
>
> What would be the advantage compared to fill ? I would guess ones and
> zeros are special because those two values are special (they can be
> defined for many types, as  neutral elements for + and *),

I couldn't find the numpy fill function, until my tiny brain realized
you meant the ndarray method:

http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.fill.html

Cheers,
Scott


From chaos.proton at gmail.com  Fri Jan 30 09:07:03 2009
From: chaos.proton at gmail.com (Grissiom)
Date: Fri, 30 Jan 2009 22:07:03 +0800
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
Message-ID: <a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>

On Fri, Jan 30, 2009 at 21:54, Scott Sinclair
<scott.sinclair.za at gmail.com>wrote:

> > 2009/1/30 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> > Neal Becker wrote:
> >> A nit, but it would be nice if 'ones' could fill with a value other than
> 1.
> >>
> >> Maybe an optional val= keyword?
> >
> > What would be the advantage compared to fill ? I would guess ones and
> > zeros are special because those two values are special (they can be
> > defined for many types, as  neutral elements for + and *),
>
> I couldn't find the numpy fill function, until my tiny brain realized
> you meant the ndarray method:
>
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.fill.html
>
> Cheers,
> Scott
>

Is fill function has any advantage over ones(size)*x ?

-- 
Cheers,
Grissiom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/2e4c9e50/attachment.html>

From sturla at molden.no  Fri Jan 30 09:11:03 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 15:11:03 +0100
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <gluunc$6t5$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>
Message-ID: <49830A77.3070507@molden.no>

On 1/30/2009 2:18 PM, Neal Becker wrote:
> A nit, but it would be nice if 'ones' could fill with a value other than 1.
> 
> Maybe an optional val= keyword?
> 

I am -1 on this. Ones should fill with ones, zeros should fill with 
zeros. Anything else is counter-intuitive. Calling numpy.ones to fill 
with fives makes no sense to me. But I would be +1 on having a function 
called numpy.values or numpy.fill that would create and fill an ndarray 
with arbitrary values.


S.M.


From matthieu.brucher at gmail.com  Fri Jan 30 09:12:21 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 30 Jan 2009 15:12:21 +0100
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
Message-ID: <e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>

> Is fill function has any advantage over ones(size)*x ?

No intermediate array (inplace) ?

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From sturla at molden.no  Fri Jan 30 09:16:25 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 15:16:25 +0100
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
References: <gluunc$6t5$1@ger.gmane.org>	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
Message-ID: <49830BB9.8040504@molden.no>

On 1/30/2009 3:07 PM, Grissiom wrote:

> Is fill function has any advantage over ones(size)*x ?

You avoid filling with ones, all the multiplications and creating an 
temporary array. It can be done like this:

a = empty(size)
a[:] = x

Which would be slightly faster and more memory efficient.


S.M.


From ndbecker2 at gmail.com  Fri Jan 30 09:22:46 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 30 Jan 2009 09:22:46 -0500
Subject: [Numpy-discussion] minor improvment to ones
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
Message-ID: <glv2fn$itt$1@ger.gmane.org>

Right now there are 2 options to create an array of constant value:

1) empty (size); fill (val)

2) ones (size) * val

1 has disadvantage of not being an expression, so can't be an arg to a 
function call.  Also probably slower than create+fill @ same time

2 is probably slower than create+fill @ same time

Now what would be _really_ cool is a special array type that would represent 
a constant array without wasting memory.  boost::ublas, for example, has 
this feature.





From gael.varoquaux at normalesup.org  Fri Jan 30 09:28:42 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 30 Jan 2009 15:28:42 +0100
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <glv2fn$itt$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org>
Message-ID: <20090130142842.GB23594@phare.normalesup.org>

On Fri, Jan 30, 2009 at 09:22:46AM -0500, Neal Becker wrote:
> Now what would be _really_ cool is a special array type that would represent 
> a constant array without wasting memory.  boost::ublas, for example, has 
> this feature.

What I would actually like is some kind of sparse masked array, which
would be a generalization of what you are talking about (your case would
be the trivial case of mask=False, and a given fill value).

Ga?l


From chaos.proton at gmail.com  Fri Jan 30 10:09:07 2009
From: chaos.proton at gmail.com (Grissiom)
Date: Fri, 30 Jan 2009 23:09:07 +0800
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <49830BB9.8040504@molden.no>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<49830BB9.8040504@molden.no>
Message-ID: <a0adeea50901300709w25763820y3cd4a23de8b7c04b@mail.gmail.com>

On Fri, Jan 30, 2009 at 22:16, Sturla Molden <sturla at molden.no> wrote:

> On 1/30/2009 3:07 PM, Grissiom wrote:
>
> > Is fill function has any advantage over ones(size)*x ?
>
> You avoid filling with ones, all the multiplications and creating an
> temporary array. It can be done like this:
>
> a = empty(size)
> a[:] = x
>
> Which would be slightly faster and more memory efficient.
>
>
Hmm,  I +1 on this one. It's more pythonic ;)

-- 
Cheers,
Grissiom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/0f4797b1/attachment.html>

From rmay31 at gmail.com  Fri Jan 30 10:19:14 2009
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 30 Jan 2009 09:19:14 -0600
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <49830A77.3070507@molden.no>
References: <gluunc$6t5$1@ger.gmane.org> <49830A77.3070507@molden.no>
Message-ID: <49831A72.2060400@gmail.com>

Sturla Molden wrote:
> On 1/30/2009 2:18 PM, Neal Becker wrote:
>> A nit, but it would be nice if 'ones' could fill with a value other than 1.
>>
>> Maybe an optional val= keyword?
>>
> 
> I am -1 on this. Ones should fill with ones, zeros should fill with 
> zeros. Anything else is counter-intuitive. Calling numpy.ones to fill 
> with fives makes no sense to me. But I would be +1 on having a function 
> called numpy.values or numpy.fill that would create and fill an ndarray 
> with arbitrary values.

I completely agree here.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From sturla at molden.no  Fri Jan 30 10:35:55 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 16:35:55 +0100
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <glv2fn$itt$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org>
Message-ID: <49831E5B.8000907@molden.no>

On 1/30/2009 3:22 PM, Neal Becker wrote:

> Now what would be _really_ cool is a special array type that would represent 
> a constant array without wasting memory.

Which again is a special case of something even cooler: lazy evaluation.

This would require arrays to have immutable buffers. Then an expression 
like "a*x + y" would result in a symbolic representation of

a * x.buffer + y.buffer

Then the array could evaluate itself (it could even numexpr or a JIT 
compiler) when needed.

The scheme would be very fragile and complicated if arrays were allowed 
to have immutable data buffers like those of numpy.


Sturla Molden


From david.froger.info at gmail.com  Fri Jan 30 11:03:44 2009
From: david.froger.info at gmail.com (David Froger)
Date: Fri, 30 Jan 2009 17:03:44 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
Message-ID: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>

Hy,

My question is about reading Fortran binary file (oh no this question
again...)

Until now, I was using the unpack module like that :

def lread(f,fourBeginning,fourEnd,*tuple):
    from struct import unpack
    """Reading a Fortran binary file in litte-endian"""
    if fourBeginning: f.seek(4,1)
    for array in tuple:
        for elt in xrange(array.size):
            transpose(array).flat[elt] =
unpack(array.dtype.char,f.read(array.itemsize))[0]
    if fourEnd: f.seek(4,1)

After googling, I read that fopen and npfille was deprecated, and we should
use numpy.fromfile and ndarray.tofile, but despite of the documentaion, the
cookbook, the mailling list and google, I don't succed in making a simple
example. Considering the simple Fortran code below what is the Python script
to read the four arrrays? What about if my pc is little endian and the file
big endian?

I think it will be a good idea to put the Fortran writting-arrays code and
the Python reading-array script in the cookbook and maybe a page to help
people comming from Fortran to start with Python ?

Best,

David Froger


program makeArray

implicit none

integer,parameter:: nx=10,ny=20
real(4),dimension(nx,ny):: ux,uy,p
integer :: i,j

open(11,file='uxuyp.bin',form='unformatted')

do i = 1,nx
do j = 1,ny
   ux(i,j) = real(i*j)
   uy(i,j) = real(i)/real(j)
   p (i,j)  = real(i) + real(j)
enddo
enddo

write(11) ux,uy
write(11) p

close(11)

end program makeArray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/55ac9ebd/attachment.html>

From sturla at molden.no  Fri Jan 30 11:23:45 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 17:23:45 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
Message-ID: <49832991.1040208@molden.no>

On 1/30/2009 5:03 PM, David Froger wrote:

> I think it will be a good idea to put the Fortran writting-arrays code 
> and the Python reading-array script in the cookbook and maybe a page to 
> help people comming from Fortran to start with Python ?

If you want to be completely safe, read the file in Fortran, then send 
it as an array to Python (use f2py). Aside from that, assuming your 
compiler only writes the raw bytes in Fortran order to the file:

If you have

real(4),dimension(nx,ny):: ux

in Fortran, and writes ux to disk. It could be retrieved like this in NumPy:

ux = np.fromfile(nx*ny, dtype=np.float32).view((nx,ny), order='F')

assuming real(kind=4) is equivalent to np.float32.


Sturla Molden


From ndbecker2 at gmail.com  Fri Jan 30 11:26:00 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 30 Jan 2009 11:26 -0500
Subject: [Numpy-discussion] concatenate trouble
Message-ID: <glv9mp$fo7$1@ger.gmane.org>

What's the problem here?

 print np.concatenate (np.ones (10, dtype=complex), np.zeros (10, 
dtype=complex))
TypeError: only length-1 arrays can be converted to Python scalars





From sturla at molden.no  Fri Jan 30 11:27:08 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 17:27:08 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <49832991.1040208@molden.no>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<49832991.1040208@molden.no>
Message-ID: <49832A5C.6040000@molden.no>

On 1/30/2009 5:23 PM, Sturla Molden wrote:

> ux = np.fromfile(nx*ny, dtype=np.float32).view((nx,ny), order='F')

oops.. this should be

ux = np.fromfile(file, count=nx*ny, dtype=np.float32).view((nx,ny),
           order='F')



S.M.


From cimrman3 at ntc.zcu.cz  Fri Jan 30 11:28:59 2009
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 30 Jan 2009 17:28:59 +0100
Subject: [Numpy-discussion] concatenate trouble
In-Reply-To: <glv9mp$fo7$1@ger.gmane.org>
References: <glv9mp$fo7$1@ger.gmane.org>
Message-ID: <49832ACB.4000109@ntc.zcu.cz>

Neal Becker wrote:
> What's the problem here?
> 
>  print np.concatenate (np.ones (10, dtype=complex), np.zeros (10, 
> dtype=complex))
> TypeError: only length-1 arrays can be converted to Python scalars

You should enclose the arrays you concatenate into a tuple: 
np.concatenate((a,b)).

r.


From sturla at molden.no  Fri Jan 30 11:31:57 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 17:31:57 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <49832A5C.6040000@molden.no>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>	<49832991.1040208@molden.no>
	<49832A5C.6040000@molden.no>
Message-ID: <49832B7D.80607@molden.no>

On 1/30/2009 5:27 PM, Sturla Molden wrote:
> On 1/30/2009 5:23 PM, Sturla Molden wrote:
> 
>> ux = np.fromfile(nx*ny, dtype=np.float32).view((nx,ny), order='F')
> 
> oops.. this should be
> 
> ux = np.fromfile(file, count=nx*ny, dtype=np.float32).view((nx,ny),
>            order='F')

fu*k


ux = np.fromfile(file, count=nx*ny,
     dtype=np.float32).reshape((nx,ny), order='F')


Sorry for the previous typos, it's Friday and soon weekend...



Sturla Molden


From gary.pajer at gmail.com  Fri Jan 30 12:20:13 2009
From: gary.pajer at gmail.com (Gary Pajer)
Date: Fri, 30 Jan 2009 12:20:13 -0500
Subject: [Numpy-discussion] Data file format choice.
Message-ID: <88fe22a0901300920m6eb1153at49c89b0227b3c5ce@mail.gmail.com>

It's time for me to select a data format.

My data are (more or less) spectra ( a couple of thousand samples), six
channels, each channel running around 10 Hz, collecting for a minute or so.
Plus all the settings on the instrument.

I don't see any significant differences between netCDF4 and HDF5.
Similarly, I don't see significant differences between pytables and h5py.
Does one play better with numpy?  What are the best numpy solutions for
netCDF4?

Can anyone provide thoughts, pros and cons, etc, that I can mull over?

-gary
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/9694c7f9/attachment.html>

From Chris.Barker at noaa.gov  Fri Jan 30 12:24:14 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 30 Jan 2009 09:24:14 -0800
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <49831E5B.8000907@molden.no>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org> <49831E5B.8000907@molden.no>
Message-ID: <498337BE.7020209@noaa.gov>

> On 1/30/2009 3:22 PM, Neal Becker wrote:
> 
>> Now what would be _really_ cool is a special array type that would represent 
>> a constant array without wasting memory.

Can't you do that with scary stride tricks? I think I remember some 
discussion of this a while back.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From rmay31 at gmail.com  Fri Jan 30 12:26:16 2009
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 30 Jan 2009 11:26:16 -0600
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <498337BE.7020209@noaa.gov>
References: <gluunc$6t5$1@ger.gmane.org>	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>	<glv2fn$itt$1@ger.gmane.org>
	<49831E5B.8000907@molden.no> <498337BE.7020209@noaa.gov>
Message-ID: <49833838.30909@gmail.com>

Christopher Barker wrote:
>> On 1/30/2009 3:22 PM, Neal Becker wrote:
>>
>>> Now what would be _really_ cool is a special array type that would represent 
>>> a constant array without wasting memory.
> 
> Can't you do that with scary stride tricks? I think I remember some 
> discussion of this a while back.

I think that's right, but at that point, what gain is that over using a regular
constant and relying on numpy's broadcasting?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From Chris.Barker at noaa.gov  Fri Jan 30 12:30:34 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 30 Jan 2009 09:30:34 -0800
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <49832991.1040208@molden.no>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<49832991.1040208@molden.no>
Message-ID: <4983393A.5010702@noaa.gov>

> If you want to be completely safe, read the file in Fortran, then send 
> it as an array to Python (use f2py). Aside from that, assuming your 
> compiler only writes the raw bytes in Fortran order to the file:

Careful -- the last time I read a Fortran-=written binary file, I found 
that the various structures (is that what you call them in Fortran?) 
were padded with I think 4 bytes.

I did it by reading through the file bit by bit and parsing it out with 
the struct module. Not very efficient, but easy to control.

In that case, there were a bunch of tings written, where a few values 
described the size of the next array, then the array, etc. If you've got 
a single array, it'll be easier.

You might start that way, and then, when you've got it figured out, 
translate to fromfile().

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From raik.gruenberg at crg.es  Fri Jan 30 12:47:16 2009
From: raik.gruenberg at crg.es (Raik Gruenberg)
Date: Fri, 30 Jan 2009 18:47:16 +0100
Subject: [Numpy-discussion] puzzle: generate index with many ranges
Message-ID: <49833D24.7060705@crg.es>

Hi there,

perhaps someone has a bright idea for this one:

I want to concatenate ranges of numbers into a single array (for indexing). So I
have generated an array "a" with starting positions, for example:

a = [4, 0, 11]

I have an array b with stop positions:

b = [11, 4, 15]

and I would like to generate an index array that takes 4..11, then 0..4, then
11..15.

In reality, a and b have 10000+ elements and the arrays to be "sliced" are very
large so I want to avoid any for loops etc. Any idea how this could be done? I
thought some combination of *repeat* and adding of *arange* should do the trick
but just cannot nail it down.

Thanks in advance for any hints!

Greetings,
Raik




From david.froger.info at gmail.com  Fri Jan 30 12:59:02 2009
From: david.froger.info at gmail.com (David Froger)
Date: Fri, 30 Jan 2009 18:59:02 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <4983393A.5010702@noaa.gov>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<49832991.1040208@molden.no> <4983393A.5010702@noaa.gov>
Message-ID: <dcba39620901300959x6433d15ch36d3a73743de96fd@mail.gmail.com>

Thank Sturla and Christopher,

yes, with the Fortran code :

!=========================================
program makeArray

implicit none

integer,parameter:: nx=2,ny=5
real(4),dimension(nx,ny):: ux,uy,p
integer :: i,j

open(11,file='uxuyp.bin',form='unformatted')

do i = 1,nx
do j = 1,ny
   ux(i,j) = 100  + j+(i-1)*10
   uy(i,j) = 200. + j+(i-1)*10
   p (i,j) = 300. + j+(i-1)*10
enddo
enddo

write(11) ux,uy
write(11) p

close(11)

write(*,*) 'ux='
do i = 1,nx
   write(*,*) ( ux(i,j) , j =1,ny)
enddo
write(*,*)

write(*,*) 'uy='
do i = 1,nx
   write(*,*) ( uy(i,j) , j =1,ny)
enddo
write(*,*)

write(*,*) 'p='
do i = 1,nx
   write(*,*) ( p(i,j) , j =1,ny)
enddo
write(*,*)

end program makeArray
!=========================================

the size of the 'uxuyp.bin' file is:

  4bytes + ux_bytes + uy_bytes + 4bytes
+ 4bytes + p_bytes + 4bytes

= 4 + nx*ny*4 + nx*ny*4 + 4 + 4 +nx*ny*4 + 4 =  136 bytes


the arrays are :

 ux=
   101.00000       102.00000       103.00000       104.00000       105.00000
   111.00000       112.00000       113.00000       114.00000       115.00000

 uy=
   201.00000       202.00000       203.00000       204.00000       205.00000
   211.00000       212.00000       213.00000       214.00000       215.00000

 p=
   301.00000       302.00000       303.00000       304.00000       305.00000
   311.00000       312.00000       313.00000       314.00000       315.00000


and with the Python script :

#===============================================
import numpy as np

nx,ny = 2,5

fourBytes = np.fromfile('uxuyp.bin', count=1, dtype=np.float32)
ux = np.fromfile('uxuyp.bin', count=nx*ny,
dtype=np.float32).reshape((nx,ny), order='F')

print ux
#===============================================

I get :

[[  1.12103877e-43   1.11000000e+02   1.12000000e+02   1.13000000e+02
    1.14000000e+02]
 [  1.01000000e+02   1.02000000e+02   1.03000000e+02   1.04000000e+02
    1.05000000e+02]]


this function do the trick, but is it optimized?

#===============================================
def lread(f,fourBeginning,fourEnd,*tuple):
    from struct import unpack
    """Reading a Fortran binary file in litte-endian"""

    if fourBeginning: f.seek(4,1)
    for array in tuple:
        for elt in xrange(array.size):
            transpose(array).flat[elt] =
unpack(array.dtype.char,f.read(array.itemsize))[0]
    if fourEnd: f.seek(4,1)
#===============================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/9f0babaf/attachment.html>

From Jim.Vickroy at noaa.gov  Fri Jan 30 13:01:15 2009
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Fri, 30 Jan 2009 11:01:15 -0700
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <49833D24.7060705@crg.es>
References: <49833D24.7060705@crg.es>
Message-ID: <4983406B.60301@noaa.gov>

Raik Gruenberg wrote:
> Hi there,
>
> perhaps someone has a bright idea for this one:
>
> I want to concatenate ranges of numbers into a single array (for indexing). So I
> have generated an array "a" with starting positions, for example:
>
> a = [4, 0, 11]
>
> I have an array b with stop positions:
>
> b = [11, 4, 15]
>
> and I would like to generate an index array that takes 4..11, then 0..4, then
> 11..15.
>   
Does this help? 

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit 
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 >>> a = [4, 0, 11]
 >>> b = [11, 4, 15]
 >>> zip(a,b)
[(4, 11), (0, 4), (11, 15)]
 >>>

Apologies if I'm stating the obvious.

-- jv
> In reality, a and b have 10000+ elements and the arrays to be "sliced" are very
> large so I want to avoid any for loops etc. Any idea how this could be done? I
> thought some combination of *repeat* and adding of *arange* should do the trick
> but just cannot nail it down.
>
> Thanks in advance for any hints!
>
> Greetings,
> Raik
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From sebastian.walter at gmail.com  Fri Jan 30 13:03:00 2009
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Fri, 30 Jan 2009 19:03:00 +0100
Subject: [Numpy-discussion] using numpy functions on an array of objects
Message-ID: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>

Hey,

What is  the best solution to get this code working?
Anyone a good idea?

------------------ test.py -----------------------------------------------
import numpy
import numpy.linalg

class afloat:
	def __init__(self,x):
		self.x = x

	def __add__(self,rhs):
		return self.x + rhs.x

	def sin(self):
		return numpy.sin(self.x)

	def inv(self):
		return numpy.linalg.inv(self.x)

	def trace(self):
		return 0

y = afloat(numpy.eye(3))
z = afloat(numpy.ones(3))
print y + z # works
print numpy.sin(y) # works
print numpy.trace(y) # doesn't work...???
print numpy.linalg.inv(y)  # doesn't work ...???

-------------------- end test.py --------------------------------


=== Explanation why I need that ===

I have the following problem. I need to do numerical calculations on
generalized versions of real numbers.
In particular with  truncated Taylor polynomials. I've implemented
that as a class that I called TC.
To define what the multiplication of two Taylor polynomials is I use
operator overloading.
Additionally, I need to compute sine, cosine, exp, etc. of Taylor polynomials.

For that, I can use the numpy functions. Numpy is apparently smart
enough to call the member function sin(self) of my class afloat
when it realizes that the argument of numpy.sin is not a known type.

This is really great. However, some functions are not as smart: Among them
trace
inv
dot


As a workaround, i could this:

def inv(X):
	if X.__class__ == afloat:
		return X.inv()
	else:
		return numpy.linalg.inv(X)

This is somewhat OK, but I'd like to use already existing Python code
that uses Numpy internally.
So I have to rely that numpy.inv(X) calls X.inv() when X is object of my class.


best regards,
Sebastian


From raik.gruenberg at crg.es  Fri Jan 30 13:11:10 2009
From: raik.gruenberg at crg.es (Raik Gruenberg)
Date: Fri, 30 Jan 2009 19:11:10 +0100
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <4983406B.60301@noaa.gov>
References: <49833D24.7060705@crg.es> <4983406B.60301@noaa.gov>
Message-ID: <498342BE.4080002@crg.es>

Jim Vickroy wrote:
> Raik Gruenberg wrote:
>> Hi there,
>>
>> perhaps someone has a bright idea for this one:
>>
>> I want to concatenate ranges of numbers into a single array (for indexing). So I
>> have generated an array "a" with starting positions, for example:
>>
>> a = [4, 0, 11]
>>
>> I have an array b with stop positions:
>>
>> b = [11, 4, 15]
>>
>> and I would like to generate an index array that takes 4..11, then 0..4, then
>> 11..15.
>>   
> Does this help? 
> 
> Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit 
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> a = [4, 0, 11]
>  >>> b = [11, 4, 15]
>  >>> zip(a,b)
> [(4, 11), (0, 4), (11, 15)]
>  >>>

Mhm, I got this far. But how do I get from here to a single index array

[ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ?

Greetings
Raik

> 



> Apologies if I'm stating the obvious.
> 
> -- jv
>> In reality, a and b have 10000+ elements and the arrays to be "sliced" are very
>> large so I want to avoid any for loops etc. Any idea how this could be done? I
>> thought some combination of *repeat* and adding of *arange* should do the trick
>> but just cannot nail it down.
>>
>> Thanks in advance for any hints!
>>
>> Greetings,
>> Raik
>>
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>   
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> 

-- 
________________________________

Dr. Raik Gruenberg
http://www.raiks.de/contact.html
________________________________


From rmay31 at gmail.com  Fri Jan 30 13:11:57 2009
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 30 Jan 2009 12:11:57 -0600
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <dcba39620901300959x6433d15ch36d3a73743de96fd@mail.gmail.com>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>	<49832991.1040208@molden.no>
	<4983393A.5010702@noaa.gov>
	<dcba39620901300959x6433d15ch36d3a73743de96fd@mail.gmail.com>
Message-ID: <498342ED.4060907@gmail.com>

David Froger wrote:
> import numpy as np
> 
> nx,ny = 2,5
> 
> fourBytes = np.fromfile('uxuyp.bin', count=1, dtype=np.float32)
> ux = np.fromfile('uxuyp.bin', count=nx*ny,
> dtype=np.float32).reshape((nx,ny), order='F')
> 
> print ux
> #===============================================
> 
> I get :
> 
> [[  1.12103877e-43   1.11000000e+02   1.12000000e+02   1.13000000e+02
>     1.14000000e+02]
>  [  1.01000000e+02   1.02000000e+02   1.03000000e+02   1.04000000e+02
>     1.05000000e+02]]
> 
> 
> this function do the trick, but is it optimized?
> 
> #===============================================
> def lread(f,fourBeginning,fourEnd,
> *tuple):
>     from struct import unpack
>     """Reading a Fortran binary file in litte-endian"""
> 
>     if fourBeginning: f.seek(4,1)
>     for array in tuple:
>         for elt in xrange(array.size):
>             transpose(array).flat[elt] =
> unpack(array.dtype.char,f.read(array.itemsize))[0]
>     if fourEnd: f.seek(4,1)
> #===============================================

I'm not sure about whether or not its optimized, but I can tell you that the
"mystery" 4 bytes are the number of bytes it that wrote out followed by that
number of bytes of data.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From jswhit at fastmail.fm  Fri Jan 30 12:31:25 2009
From: jswhit at fastmail.fm (Jeff Whitaker)
Date: Fri, 30 Jan 2009 10:31:25 -0700
Subject: [Numpy-discussion] Data file format choice.
In-Reply-To: <88fe22a0901300920m6eb1153at49c89b0227b3c5ce@mail.gmail.com>
References: <88fe22a0901300920m6eb1153at49c89b0227b3c5ce@mail.gmail.com>
Message-ID: <4983396D.6030903@fastmail.fm>

Gary Pajer wrote:
> It's time for me to select a data format.
>
> My data are (more or less) spectra ( a couple of thousand samples), 
> six channels, each channel running around 10 Hz, collecting for a 
> minute or so. Plus all the settings on the instrument.
>
> I don't see any significant differences between netCDF4 and HDF5.   
Gary:  netCDF4 is just a thin wrapper on top of HDF5 1.8 - think of it 
as a higher level API.
> Similarly, I don't see significant differences between pytables and 
> h5py.  Does one play better with numpy?  
pytables has been around longer and is well-tested, has nice pythonic 
features, but files you write with it may not be readable by C or 
fortran clients.  h5py works only with python 2.5/2.6, and writes 
'vanilla' hdf5 files readable by anybody.
> What are the best numpy solutions for netCDF4?

There's only one that I know of - http://code.google.com/p/netcdf4-python.

-Jeff

-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/PSD  R/PSD1        Email  : Jeffrey.S.Whitaker at noaa.gov
325 Broadway                Office : Skaggs Research Cntr 1D-113
Boulder, CO, USA 80303-3328 Web    : http://tinyurl.com/5telg



From Jim.Vickroy at noaa.gov  Fri Jan 30 13:37:41 2009
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Fri, 30 Jan 2009 11:37:41 -0700
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <498342BE.4080002@crg.es>
References: <49833D24.7060705@crg.es> <4983406B.60301@noaa.gov>
	<498342BE.4080002@crg.es>
Message-ID: <498348F5.50506@noaa.gov>

Raik Gruenberg wrote:
> Jim Vickroy wrote:
>   
>> Raik Gruenberg wrote:
>>     
>>> Hi there,
>>>
>>> perhaps someone has a bright idea for this one:
>>>
>>> I want to concatenate ranges of numbers into a single array (for indexing). So I
>>> have generated an array "a" with starting positions, for example:
>>>
>>> a = [4, 0, 11]
>>>
>>> I have an array b with stop positions:
>>>
>>> b = [11, 4, 15]
>>>
>>> and I would like to generate an index array that takes 4..11, then 0..4, then
>>> 11..15.
>>>   
>>>       
>> Does this help? 
>>
>> Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit 
>> (Intel)] on win32
>> Type "help", "copyright", "credits" or "license" for more information.
>>  >>> a = [4, 0, 11]
>>  >>> b = [11, 4, 15]
>>  >>> zip(a,b)
>> [(4, 11), (0, 4), (11, 15)]
>>  >>>
>>     
>
> Mhm, I got this far. But how do I get from here to a single index array
>
> [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ?
>
>   
not sure I understand your goal ... is this what you want:
 >>> [range(i,j) for i,j in zip(a,b)]
[[4, 5, 6, 7, 8, 9, 10], [0, 1, 2, 3], [11, 12, 13, 14]]
 >>>
> Greetings
> Raik
>
>   
>
>
>
>   
>> Apologies if I'm stating the obvious.
>>
>> -- jv
>>     
>>> In reality, a and b have 10000+ elements and the arrays to be "sliced" are very
>>> large so I want to avoid any for loops etc. Any idea how this could be done? I
>>> thought some combination of *repeat* and adding of *arange* should do the trick
>>> but just cannot nail it down.
>>>
>>> Thanks in advance for any hints!
>>>
>>> Greetings,
>>> Raik
>>>
>>>
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>   
>>>       
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>     
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/58426c2e/attachment.html>

From pgmdevlist at gmail.com  Fri Jan 30 13:44:46 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 30 Jan 2009 13:44:46 -0500
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <498342BE.4080002@crg.es>
References: <49833D24.7060705@crg.es> <4983406B.60301@noaa.gov>
	<498342BE.4080002@crg.es>
Message-ID: <7D622E3E-D52F-43F1-9379-1531A44792F8@gmail.com>


On Jan 30, 2009, at 1:11 PM, Raik Gruenberg wrote:

>
> Mhm, I got this far. But how do I get from here to a single index  
> array
>
> [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ?

np.concatenate([np.arange(aa,bb) for (aa,bb) in zip(a,b)])


From raik.gruenberg at crg.es  Fri Jan 30 13:53:57 2009
From: raik.gruenberg at crg.es (Raik Gruenberg)
Date: Fri, 30 Jan 2009 19:53:57 +0100
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <7D622E3E-D52F-43F1-9379-1531A44792F8@gmail.com>
References: <49833D24.7060705@crg.es>
	<4983406B.60301@noaa.gov>	<498342BE.4080002@crg.es>
	<7D622E3E-D52F-43F1-9379-1531A44792F8@gmail.com>
Message-ID: <49834CC5.4080509@crg.es>

Pierre GM wrote:
> On Jan 30, 2009, at 1:11 PM, Raik Gruenberg wrote:
> 
>> Mhm, I got this far. But how do I get from here to a single index  
>> array
>>
>> [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ?
> 
> np.concatenate([np.arange(aa,bb) for (aa,bb) in zip(a,b)])

exactly! Now, the question was, is there a way to do this only using numpy 
functions (sum, repeat, ...), that means without any python "for" loop?

Sorry about being so insistent on this one but, in my experience, eliminating 
those for loops makes a huge difference in terms of speed. The zip is probably 
also quite costly on a very large data set.

Thanks!
Raik

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> 

-- 
________________________________

Dr. Raik Gruenberg
http://www.raiks.de/contact.html
________________________________


From f.yw at hotmail.com  Fri Jan 30 13:58:21 2009
From: f.yw at hotmail.com (frank wang)
Date: Fri, 30 Jan 2009 11:58:21 -0700
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl> 
	<3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com> 
Message-ID: <BLU105-W51137769905200294A75F789C60@phx.gbl>


I have created a test example for the question using for loop and hope someone can help me to get fast solution. My data set is about 2000000 data.
 
However, I have the problem to run the code, the Out[i]=cnstl[j] line gives me error says:
 
In [107]: Out[0]=cnstl[0]---------------------------------------------------------------------------TypeError                                 Traceback (most recent call last)
C:\Frank_share\qamslicer.py in <module>()----> 1      2      3      4      5
TypeError: can't convert complex to float; use abs(z)
In [108]: cnstl.dtypeOut[108]: dtype('complex128')
I do not know why that my data is complex128 already. Can anyone help to figure why?
 
Thanks
 
Frank
 
from numpy import *a = arange(-15,16,2)cnstl=a.reshape(16,1)+1j*acnstl=cnstl.reshape(256,1)
X = array([1.4 + 1j*2.7, -4.9 + 1j*8.3])
Out = array(X)error =array(X)for i in xrange(2):    for j in xrange(256):        a0 = real(X[i]) < (real(cnstl[j])+1)        a1 = real(X[i]) > (real(cnstl[j])-1)        a2 = imag(X[i]) > (imag(cnstl[j])-1)        a3 = imag(X[i]) < (imag(cnstl[j])+1) if (a0 & a1 & a2 &a3):            Out[i] = cnstl[j]            error[i] = X[i] - cnstl[j]

From: f.yw at hotmail.comTo: numpy-discussion at scipy.orgSubject: RE: [Numpy-discussion] help on fast slicing on a gridDate: Wed, 28 Jan 2009 23:28:47 -0700

Hi, Bob, Thanks for your help.  I am sorry for my type error. qam array is the X array in my example. cntl is a complex array contains the point (x,y) axises. I will try to make a workable example. Also I will try to find out the zeros_like function. However, I guess that zeros_like(X) will create an array the same size as X. It it is. Then the two line Out=X and error=X should be Out=zeros_like(X) and error=zeros(X). Also, can where command handel the logic command? aa = np.where((real(X)<real(cnstl[j])+1) & (real(X)>real(cnstl[j])-1) & (imag(X)<imag(cnstl[j])+1) & (imag(X)>imag(cnstl[j]-1)) For example, cntl[j]=3+1j*5, then the where command is the same as: aa = np.where((real(X)<4) & (real(X)>2 )& (imag(X)<6) & (imag(X)>4)) Thanks Frank> Date: Thu, 29 Jan 2009 00:15:48 -0600> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] help on fast slicing on a grid> > On Thu, Jan 29, 2009 at 00:09, frank wang <f.yw at hotmail.com> wrote:> > Here is the for loop that I am think about. Also, I do not know whether the> > where commands can handle the complicated logic.> > The where command basically find the data in the square around the point> > cnstl[j].> > cnstl is a 2D array from your previous description.> > > Let the data array is qam with size N> > I don't see qam anywhere. Did you mean X?> > > Out = X> > error = X> > Don't you want something like zeros_like(X) for these?> > > for i in arange(N):> > for j in arange(L):> > aa = np.where((real(X)<real(cnstl[j])+1) &> > (real(X)>real(cnstl[j])-1) & (imag(X)<imag(cnstl[j])+1) &> > (imag(X)>imag(cnstl[j]-1))> > Out[aa]=cnstl[j]> > error[aa]=abs(X)**2 - abs(cnstl[j])**2> > I'm still confused. Can you show me a complete, working script with> possibly fake data?> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion

Windows Live?: E-mail. Chat. Share. Get more ways to connect. Check it out.
_________________________________________________________________
Hotmail? goes where you go. On a PC, on the Web, on your phone. 
http://www.windowslive-hotmail.com/learnmore/versatility.aspx#mobile?ocid=TXT_TAGHM_WL_HM_versatility_121208 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/1f8b0858/attachment.html>

From pgmdevlist at gmail.com  Fri Jan 30 14:06:26 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 30 Jan 2009 14:06:26 -0500
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <49834CC5.4080509@crg.es>
References: <49833D24.7060705@crg.es>
	<4983406B.60301@noaa.gov>	<498342BE.4080002@crg.es>
	<7D622E3E-D52F-43F1-9379-1531A44792F8@gmail.com>
	<49834CC5.4080509@crg.es>
Message-ID: <78A727A0-B07B-4086-9126-7805ACFBE242@gmail.com>


On Jan 30, 2009, at 1:53 PM, Raik Gruenberg wrote:

> Pierre GM wrote:
>> On Jan 30, 2009, at 1:11 PM, Raik Gruenberg wrote:
>>
>>> Mhm, I got this far. But how do I get from here to a single index
>>> array
>>>
>>> [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ?
>>
>> np.concatenate([np.arange(aa,bb) for (aa,bb) in zip(a,b)])
>
> exactly! Now, the question was, is there a way to do this only using  
> numpy
> functions (sum, repeat, ...), that means without any python "for"  
> loop?

Can't really see it right now. Make np.arange(max(b)) and take the  
slices you need ? But you still have to look in 2 arrays to find the  
beginning and end of slices, so...


> Sorry about being so insistent on this one but, in my experience,  
> eliminating
> those for loops makes a huge difference in terms of speed. The zip  
> is probably
> also quite costly on a very large data set.

yeah, but it's in a list comprehension, which may make things a tad  
faster. If you prefer, use itertools.izip instead of zip, but I wonder  
where the advantage would be. Anyway, are you sure this particular  
part is your bottleneck ? You know the saying about premature  
optimization...


From Chris.Barker at noaa.gov  Fri Jan 30 14:18:10 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 30 Jan 2009 11:18:10 -0800
Subject: [Numpy-discussion] using numpy functions on an array of objects
In-Reply-To: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>
References: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>
Message-ID: <49835272.1060704@noaa.gov>

I think you want to subclass an ndarray here. It's a bit tricky to so, 
but if you look in the wiki and these mailing list archives, you'll find 
advise on how to do it.


-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From faltet at pytables.org  Fri Jan 30 14:42:23 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 30 Jan 2009 20:42:23 +0100
Subject: [Numpy-discussion] Data file format choice.
In-Reply-To: <4983396D.6030903@fastmail.fm>
References: <88fe22a0901300920m6eb1153at49c89b0227b3c5ce@mail.gmail.com>
	<4983396D.6030903@fastmail.fm>
Message-ID: <200901302042.23334.faltet@pytables.org>

A Friday 30 January 2009, Jeff Whitaker escrigu?:
> Gary Pajer wrote:
> > It's time for me to select a data format.
> >
> > My data are (more or less) spectra ( a couple of thousand samples),
> > six channels, each channel running around 10 Hz, collecting for a
> > minute or so. Plus all the settings on the instrument.
> >
> > I don't see any significant differences between netCDF4 and HDF5.
>
> Gary:  netCDF4 is just a thin wrapper on top of HDF5 1.8 - think of
> it as a higher level API.
>
> > Similarly, I don't see significant differences between pytables and
> > h5py.  Does one play better with numpy?
>
> pytables has been around longer and is well-tested, has nice pythonic
> features, but files you write with it may not be readable by C or
> fortran clients.

Just to be clear.  PyTables only will write pickled objects on file if 
it is not possible to reasonably represent them as native HDF5 objects.  
But, if you try to save NumPy objects or regular Python scalars they 
are effectively written as native HDF5 objects (see [1]).

Regarding a comparison with h5py (disclaimer: I'm the main author of 
PyTables), I'd say that h5py is thought to have a direct map with NumPy 
array capabilities, but doesn't try to go further.  Also, it is worth 
to note that h5py offers access to the low-level HDF5 functions, which 
can be interesting if you want to get deeper into HDF5 intrincacies, 
which can be great for some users.  

On his hand, PyTables doesn't try to go this low-level and, besides 
supporting general NumPy objects, it is more focused on implementing 
advanced features that are normally only available in database-oriented 
approaches, like enumerated types, flexible query iterators for tables 
(on-disk equivalent to recarrays), indexing (only Pro version), do/undo 
features or natural naming (for an enhanced interactive experience).  
PyTables also tries hard to be a high performance interface to HDF5, 
implementing niceties like internal LRU caches for nodes, automatic 
chunksizes for the datasets or making use of numexpr internally so as 
to accelerate queries to a maximum.

Finally, and although h5py is relatively recent, I'm really impressed by 
the work that Andrew has already done, and in fact, I'm looking forward 
to backport some of the h5py features (like general NumPy-like fancy 
indexing capabilities) to PyTables.  At any rate, it is clear that the 
binomial h5py/PyTables will benefit users, with the only handicap that 
they have to choose their preferred API to HDF5 (or they can use both, 
which could be really a lot of fun ;-).  NetCDF4-based interfaces are 
also probably a good approach and, as it is based in HDF5, the 
compatibility is ensured.

HTH,

[1] http://www.pytables.org/docs/manual/ch04.html#id2553542

-- 
Francesc Alted


From sturla at molden.no  Fri Jan 30 15:06:11 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 30 Jan 2009 21:06:11 +0100 (CET)
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <4983393A.5010702@noaa.gov>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<49832991.1040208@molden.no> <4983393A.5010702@noaa.gov>
Message-ID: <6df541a5c8d6ecf26b6e38f404958401.squirrel@webmail.uio.no>


> Careful -- the last time I read a Fortran-=written binary file, I found
> that the various structures (is that what you call them in Fortran?)
> were padded with I think 4 bytes.

That is precisely why I suggested using f2py. If you let Fortran read the
file (be careful to the same compiler!), it is easy to pass the data on to
Python.

Otherwise, you will have to figure out how your Fortran program writes the
file. I.e. what padding, metainformation, etc. that are used. If you
switch Fortran compiler, or even compiler version from the same vendor,
you must start over again. On the other hand, an f2py'd solution just
requires a recompile.

For my own work, I just makes sure NEVER to do any I/O in Fortran! It is
asking for trouble. I leave the I/O to Python or C, where it belongs. That
way I know what data are written and what data are read.

S. M.



From faltet at pytables.org  Fri Jan 30 15:30:59 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 30 Jan 2009 21:30:59 +0100
Subject: [Numpy-discussion] NumPy distutils limitation?
Message-ID: <200901302130.59483.faltet@pytables.org>

Hi,

Gregor and I are trying to give support for Intel's VML (Vector
Mathematical Library) in numexpr.  For this, we are trying to make use
of the weaponery in NumPy's distutils so as to be able to discover
where the MKL (the package that contains VML) is located.  The
libraries that we need to link with are: mkl_gf_lp64, mkl_gnu_thread,
mkl_core *and* iomp5 (Intel's OpenMP library).

The problem is that I have installed MKL as part as the Intel compiler
for Unix.  In this setup, most of the libraries are in one place,
namely:

/opt/intel/Compiler/11.0/074/mkl/lib/em64t/

However, the OpenMP library is in another directory:

/opt/intel/Compiler/11.0/074/lib/intel64

So, I need to specify *two* directories to get the complete set of
libraries.  My first attempt was setting a site.cfg like:

[DEFAULT]
#libraries = gfortran

[mkl]
library_dirs= /opt/intel/Compiler/11.0/074/mkl/lib/em64t/:/opt/intel/Compiler/11.0/074/lib/intel64
include_dirs =  /opt/intel/Compiler/11.0/074/mkl/include/
mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core, iomp5


Unfortunately, distutils complains and says that it cannot find the
complete set of libraries:

mkl_info:
  libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found 
in /opt/intel/Compiler/11.0/074/mkl/lib/em64t/
  libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found 
in /opt/intel/Compiler/11.0/074/lib/intel64
  NOT AVAILABLE

After some debugging of the problem, it seems that distutils needs to
find *all* the required libraries in *one* single directory.  As iomp5
is on a different directory, distutils thinks that the requisites
are not fulfilled.

I've solved this by requering the iomp5 to be find in the DEFAULT
section.  Something like:

[DEFAULT]
library_dirs = /opt/intel/Compiler/11.0/074/lib/intel64
#libraries = gfortran, iomp5
libraries = iomp5

[mkl]
library_dirs = /opt/intel/Compiler/11.0/074/mkl/lib/em64t/
include_dirs =  /opt/intel/Compiler/11.0/074/mkl/include/
mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core


However, in case one would need to specify several other directories
in the DEFAULT section for finding other hypothetical necessary
libraries (like gfortran or others), we may run into the same problem 
than above.

My question is, is there an elegant way to handle this problem, or it
is a limitation of the current distutils?  If the later, it would be
nice it that could be solved in a future version, and several libraries 
can be found in *several* directories.

Thanks,

-- 
Francesc Alted


From robert.kern at gmail.com  Fri Jan 30 15:31:07 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 14:31:07 -0600
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <49833838.30909@gmail.com>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org> <49831E5B.8000907@molden.no>
	<498337BE.7020209@noaa.gov> <49833838.30909@gmail.com>
Message-ID: <3d375d730901301231n19cb4748i56f3273d492b1b3d@mail.gmail.com>

On Fri, Jan 30, 2009 at 11:26, Ryan May <rmay31 at gmail.com> wrote:
> Christopher Barker wrote:
>>> On 1/30/2009 3:22 PM, Neal Becker wrote:
>>>
>>>> Now what would be _really_ cool is a special array type that would represent
>>>> a constant array without wasting memory.
>>
>> Can't you do that with scary stride tricks? I think I remember some
>> discussion of this a while back.
>
> I think that's right, but at that point, what gain is that over using a regular
> constant and relying on numpy's broadcasting?

The filled array may be providing some of the shape information that's
not in the other arrays.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan 30 15:35:59 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 14:35:59 -0600
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <glv2fn$itt$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org>
Message-ID: <3d375d730901301235vebebc65u8b12b8ff9e656664@mail.gmail.com>

On Fri, Jan 30, 2009 at 08:22, Neal Becker <ndbecker2 at gmail.com> wrote:
> Right now there are 2 options to create an array of constant value:
>
> 1) empty (size); fill (val)
>
> 2) ones (size) * val
>
> 1 has disadvantage of not being an expression, so can't be an arg to a
> function call.

So wrap it in a function.

> Also probably slower than create+fill @ same time

Only marginally. In any case, (1) is exactly how ones() and zeros()
are implemented. I would be +1 on a patch that adds a filled()
function along the lines of ones() and zeros(), but I'm -1 on adding
this functionality to ones() or zeros().

> 2 is probably slower than create+fill @ same time
>
> Now what would be _really_ cool is a special array type that would represent
> a constant array without wasting memory.  boost::ublas, for example, has
> this feature.

In [2]: from numpy.lib.stride_tricks import as_strided

In [3]: def hollow_filled(shape, value, dtype=None):
   ...:     x = asarray(value, dtype=dtype)
   ...:     return as_strided(x, shape, [0]*len(shape))
   ...:

In [5]: hollow_filled([2,3,4], 5)
Out[5]:
array([[[5, 5, 5, 5],
        [5, 5, 5, 5],
        [5, 5, 5, 5]],

       [[5, 5, 5, 5],
        [5, 5, 5, 5],
        [5, 5, 5, 5]]])

In [6]: hollow_filled([2,3,4], 5.0)
Out[6]:
array([[[ 5.,  5.,  5.,  5.],
        [ 5.,  5.,  5.,  5.],
        [ 5.,  5.,  5.,  5.]],

       [[ 5.,  5.,  5.,  5.],
        [ 5.,  5.,  5.,  5.],
        [ 5.,  5.,  5.,  5.]]])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan 30 15:37:52 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 14:37:52 -0600
Subject: [Numpy-discussion] NumPy distutils limitation?
In-Reply-To: <200901302130.59483.faltet@pytables.org>
References: <200901302130.59483.faltet@pytables.org>
Message-ID: <3d375d730901301237l2da7f964jb14c972f5a52e0e8@mail.gmail.com>

On Fri, Jan 30, 2009 at 14:30, Francesc Alted <faltet at pytables.org> wrote:

> My question is, is there an elegant way to handle this problem, or it
> is a limitation of the current distutils?

Probably the latter.

> If the later, it would be
> nice it that could be solved in a future version, and several libraries
> can be found in *several* directories.

Patches are welcome.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan 30 15:52:27 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 14:52:27 -0600
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <BLU105-W51137769905200294A75F789C60@phx.gbl>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>
	<3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
	<BLU105-W51137769905200294A75F789C60@phx.gbl>
Message-ID: <3d375d730901301252j14628de9ye143b9d1aa3f18b5@mail.gmail.com>

On Fri, Jan 30, 2009 at 12:58, frank wang <f.yw at hotmail.com> wrote:
> I have created a test example for the question using for loop and hope
> someone can help me to get fast solution. My data set is about 2000000 data.
>
> However, I have the problem to run the code, the Out[i]=cnstl[j] line gives
> me error says:
>
> In [107]: Out[0]=cnstl[0]
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> C:\Frank_share\qamslicer.py in <module>()
> ----> 1
>       2
>       3
>       4
>       5
> TypeError: can't convert complex to float; use abs(z)
> In [108]: cnstl.dtype
> Out[108]: dtype('complex128')
>
> I do not know why that my data is complex128 already. Can anyone help to
> figure why?

It's an odd error message, certainly. The root of the problem is that
you are attempting to put a (1,)-shaped array into a scalar. You don't
want to do that.

By the way, you don't want to use the variable name Out in IPython.
It's already used to capture the output.

> Thanks
>
> Frank
>
> from numpy import *
> a = arange(-15,16,2)
> cnstl=a.reshape(16,1)+1j*a
> cnstl=cnstl.reshape(256,1)

Change that line to

  cnstl = cnstl.ravel()

> X = array([1.4 + 1j*2.7, -4.9 + 1j*8.3])
> Out = array(X)
> error =array(X)
> for i in xrange(2):
>     for j in xrange(256):
>         a0 = real(X[i]) < (real(cnstl[j])+1)
>         a1 = real(X[i]) > (real(cnstl[j])-1)
>         a2 = imag(X[i]) > (imag(cnstl[j])-1)
>         a3 = imag(X[i]) < (imag(cnstl[j])+1)
>  if (a0 & a1 & a2 &a3):
>             Out[i] = cnstl[j]
>             error[i] = X[i] - cnstl[j]

After reindenting this correctly, I get the following results:

In [22]: out
Out[22]: array([ 1.+3.j, -5.+9.j])

In [23]: error
Out[23]: array([ 0.4-0.3j,  0.1-0.7j])


Are those correct?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Jan 30 16:05:55 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 15:05:55 -0600
Subject: [Numpy-discussion] using numpy functions on an array of objects
In-Reply-To: <49835272.1060704@noaa.gov>
References: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>
	<49835272.1060704@noaa.gov>
Message-ID: <3d375d730901301305r6eb3cf11p5f4e8a0641bbf65c@mail.gmail.com>

On Fri, Jan 30, 2009 at 13:18, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> I think you want to subclass an ndarray here. It's a bit tricky to so,
> but if you look in the wiki and these mailing list archives, you'll find
> advise on how to do it.

That still won't work. numpy.linalg.inv() simply does a particular
algorithm on float and complex arrays and nothing else.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From rlw at stsci.edu  Fri Jan 30 17:20:01 2009
From: rlw at stsci.edu (Rick White)
Date: Fri, 30 Jan 2009 17:20:01 -0500
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <mailman.5.1233338404.13748.numpy-discussion@scipy.org>
References: <mailman.5.1233338404.13748.numpy-discussion@scipy.org>
Message-ID: <5A3AA485-6FDB-4F02-96C7-D77DCFBAEFF6@stsci.edu>

Here's a technique that works:

Python 2.4.2 (#5, Nov 21 2005, 23:08:11)
[GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy as np
 >>> a = np.array([0,4,0,11])
 >>> b = np.array([-1,11,4,15])
 >>> rangelen = b-a+1
 >>> cumlen = rangelen.cumsum()
 >>> c = np.arange(cumlen[-1],dtype=np.int32)
 >>> c += np.repeat(a[1:]-c[cumlen[0:-1]], rangelen[1:])
 >>> print c
[ 4  5  6  7  8  9 10 11  0  1  2  3  4 11 12 13 14 15]

The basic idea is that the difference of your desired output from a  
simple range is an array with a bunch of constant values appended  
together, and that is what repeat() does.  I'm assuming that you'll  
never have b < a.  Notice the slight ugliness of prepending the  
elements at the beginning so that the cumsum starts with zero.   
(Maybe there is a cleaner way to do that.)

This does create a second array (via the repeat) that is the same  
length as the result.  If that uses too much memory, you could break  
up the repeat and update of c into segments using a loop.  (You  
wouldn't need a loop for every a,b element -- do a bunch in each  
iteration.)

-- Rick

Raik Gruenberg wrote:

> Hi there,
>
> perhaps someone has a bright idea for this one:
>
> I want to concatenate ranges of numbers into a single array (for  
> indexing). So I
> have generated an array "a" with starting positions, for example:
>
> a = [4, 0, 11]
>
> I have an array b with stop positions:
>
> b = [11, 4, 15]
>
> and I would like to generate an index array that takes 4..11, then  
> 0..4, then
> 11..15.
>
> In reality, a and b have 10000+ elements and the arrays to be  
> "sliced" are very
> large so I want to avoid any for loops etc. Any idea how this could  
> be done? I
> thought some combination of *repeat* and adding of *arange* should  
> do the trick
> but just cannot nail it down.
>
> Thanks in advance for any hints!
>
> Greetings,
> Raik



From ndbecker2 at gmail.com  Fri Jan 30 17:32:06 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 30 Jan 2009 17:32:06 -0500
Subject: [Numpy-discussion] minor improvment to ones
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org>
	<3d375d730901301235vebebc65u8b12b8ff9e656664@mail.gmail.com>
Message-ID: <glvv56$q9c$1@ger.gmane.org>

Robert Kern wrote:

> On Fri, Jan 30, 2009 at 08:22, Neal Becker <ndbecker2 at gmail.com> wrote:
>> Right now there are 2 options to create an array of constant value:
>>
>> 1) empty (size); fill (val)
>>
>> 2) ones (size) * val
>>
>> 1 has disadvantage of not being an expression, so can't be an arg to a
>> function call.
> 
> So wrap it in a function.
> 
>> Also probably slower than create+fill @ same time
> 
> Only marginally. In any case, (1) is exactly how ones() and zeros()
> are implemented. I would be +1 on a patch that adds a filled()
> function along the lines of ones() and zeros(), but I'm -1 on adding
> this functionality to ones() or zeros().
> 
>> 2 is probably slower than create+fill @ same time
>>
>> Now what would be _really_ cool is a special array type that would
>> represent
>> a constant array without wasting memory.  boost::ublas, for example, has
>> this feature.
> 
> In [2]: from numpy.lib.stride_tricks import as_strided
> 
> In [3]: def hollow_filled(shape, value, dtype=None):
>    ...:     x = asarray(value, dtype=dtype)
>    ...:     return as_strided(x, shape, [0]*len(shape))
>    ...:
> 
> In [5]: hollow_filled([2,3,4], 5)
> Out[5]:
> array([[[5, 5, 5, 5],
>         [5, 5, 5, 5],
>         [5, 5, 5, 5]],
> 
>        [[5, 5, 5, 5],
>         [5, 5, 5, 5],
>         [5, 5, 5, 5]]])
> 
> In [6]: hollow_filled([2,3,4], 5.0)
> Out[6]:
> array([[[ 5.,  5.,  5.,  5.],
>         [ 5.,  5.,  5.,  5.],
>         [ 5.,  5.,  5.,  5.]],
> 
>        [[ 5.,  5.,  5.,  5.],
>         [ 5.,  5.,  5.,  5.],
>         [ 5.,  5.,  5.,  5.]]])
>

Where can I find doc on stride_tricks?  Nothing here:
http://docs.scipy.org/doc/numpy/search.html?q=stride_tricks&check_keywords=yes&area=default





From robert.kern at gmail.com  Fri Jan 30 17:40:29 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 16:40:29 -0600
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <glvv56$q9c$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>
	<4982FB26.1010509@ar.media.kyoto-u.ac.jp>
	<6a17e9ee0901300554j516b6005hd44c5978b92904ee@mail.gmail.com>
	<a0adeea50901300607oeaf3838ve6018bdda86aeb68@mail.gmail.com>
	<e76aa17f0901300612n7e8e3c6ap57d3cd5967bc6aca@mail.gmail.com>
	<glv2fn$itt$1@ger.gmane.org>
	<3d375d730901301235vebebc65u8b12b8ff9e656664@mail.gmail.com>
	<glvv56$q9c$1@ger.gmane.org>
Message-ID: <3d375d730901301440h50027dbdi62f48db49875ef24@mail.gmail.com>

On Fri, Jan 30, 2009 at 16:32, Neal Becker <ndbecker2 at gmail.com> wrote:
> Robert Kern wrote:
>
>> On Fri, Jan 30, 2009 at 08:22, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> Right now there are 2 options to create an array of constant value:
>>>
>>> 1) empty (size); fill (val)
>>>
>>> 2) ones (size) * val
>>>
>>> 1 has disadvantage of not being an expression, so can't be an arg to a
>>> function call.
>>
>> So wrap it in a function.
>>
>>> Also probably slower than create+fill @ same time
>>
>> Only marginally. In any case, (1) is exactly how ones() and zeros()
>> are implemented. I would be +1 on a patch that adds a filled()
>> function along the lines of ones() and zeros(), but I'm -1 on adding
>> this functionality to ones() or zeros().
>>
>>> 2 is probably slower than create+fill @ same time
>>>
>>> Now what would be _really_ cool is a special array type that would
>>> represent
>>> a constant array without wasting memory.  boost::ublas, for example, has
>>> this feature.
>>
>> In [2]: from numpy.lib.stride_tricks import as_strided
>>
>> In [3]: def hollow_filled(shape, value, dtype=None):
>>    ...:     x = asarray(value, dtype=dtype)
>>    ...:     return as_strided(x, shape, [0]*len(shape))
>>    ...:
>>
>> In [5]: hollow_filled([2,3,4], 5)
>> Out[5]:
>> array([[[5, 5, 5, 5],
>>         [5, 5, 5, 5],
>>         [5, 5, 5, 5]],
>>
>>        [[5, 5, 5, 5],
>>         [5, 5, 5, 5],
>>         [5, 5, 5, 5]]])
>>
>> In [6]: hollow_filled([2,3,4], 5.0)
>> Out[6]:
>> array([[[ 5.,  5.,  5.,  5.],
>>         [ 5.,  5.,  5.,  5.],
>>         [ 5.,  5.,  5.,  5.]],
>>
>>        [[ 5.,  5.,  5.,  5.],
>>         [ 5.,  5.,  5.,  5.],
>>         [ 5.,  5.,  5.,  5.]]])
>>
>
> Where can I find doc on stride_tricks?

Source is always the best place. as_strided is not exposed as such
since you can cause segfaults with it if you have a bug. Rather, it's
useful for devs to make tools that, once debugged, can't cause
segfaults.

> Nothing here:
> http://docs.scipy.org/doc/numpy/search.html?q=stride_tricks&check_keywords=yes&area=default

Use this search box to search the development version of the docs:

  http://docs.scipy.org/numpy/search/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david.froger.info at gmail.com  Fri Jan 30 17:53:06 2009
From: david.froger.info at gmail.com (David Froger)
Date: Fri, 30 Jan 2009 23:53:06 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <6df541a5c8d6ecf26b6e38f404958401.squirrel@webmail.uio.no>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<49832991.1040208@molden.no> <4983393A.5010702@noaa.gov>
	<6df541a5c8d6ecf26b6e38f404958401.squirrel@webmail.uio.no>
Message-ID: <dcba39620901301453n21b9221di25c5267c8ec66bcf@mail.gmail.com>

ok for f2py!

> Otherwise, you will have to figure out how your Fortran program writes the
>file. I.e. what padding, metainformation, etc. that are used. If you
>switch Fortran compiler, or even compiler version from the same vendor,
>you must start over again.

In my experience, I never had this kind of problem. I just have to convert
files between big/little endian with uswap (http://linux.die.net/man/1/uswap),
but I never see, in my experience, a Fortran program writting data
differently given the compilator.

>For my own work, I just makes sure NEVER to do any I/O in Fortran! It is
>asking for trouble. I leave the I/O to Python or C, where it belongs. That
>way I know what data are written and what data are read.

Unfortunately., binary files are mandatory in the contex I work. I use a
scientific code written in Fortran to compute fluid dynamics. Typically the
simulation is run on supercalculator and generate giga and giga of datum, so
we must use the binary format, which recquire less memory for the stockage.
Then, I like to post-trait the datum using Python and Gnuplot.py.

That's why I'm looking for a performantant, easy and 'standard'  way to read
binary Fortran files. (I think many people have the same need).

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/7dfaadd9/attachment.html>

From ellisonbg.net at gmail.com  Fri Jan 30 19:13:16 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 30 Jan 2009 16:13:16 -0800
Subject: [Numpy-discussion] numpy.distutils and shared libraries
Message-ID: <6ce0ac130901301613i53be53bs2607c6fd86cc4222@mail.gmail.com>

Hi,

2 ?'s about numpy.distutils:

1.

I am using config.add_library to build a c++ library that I will link
into some Cython extensions.  This is working fine and generating a .a
library for me.  However, I need a shared library instead.  Is this
possible with numpy.distutils or will I need something like numscons?

2.

When calling add_library, what is the difference between the depends
and headers arguments?

Thanks!

Brian


From robert.kern at gmail.com  Fri Jan 30 19:21:30 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 18:21:30 -0600
Subject: [Numpy-discussion] numpy.distutils and shared libraries
In-Reply-To: <6ce0ac130901301613i53be53bs2607c6fd86cc4222@mail.gmail.com>
References: <6ce0ac130901301613i53be53bs2607c6fd86cc4222@mail.gmail.com>
Message-ID: <3d375d730901301621l3391ea97id48ca8d674c257f4@mail.gmail.com>

On Fri, Jan 30, 2009 at 18:13, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Hi,
>
> 2 ?'s about numpy.distutils:
>
> 1.
>
> I am using config.add_library to build a c++ library that I will link
> into some Cython extensions.  This is working fine and generating a .a
> library for me.  However, I need a shared library instead.  Is this
> possible with numpy.distutils or will I need something like numscons?

numscons or you can adapt the code from OOF2 and contribute it to
numpy.distutils.

> 2.
>
> When calling add_library, what is the difference between the depends
> and headers arguments?

headers get installed via the distutils install_headers command. This
will install the headers ... somewhere. Not exactly sure.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Fri Jan 30 19:31:15 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 30 Jan 2009 16:31:15 -0800
Subject: [Numpy-discussion] numpy.distutils and shared libraries
In-Reply-To: <3d375d730901301621l3391ea97id48ca8d674c257f4@mail.gmail.com>
References: <6ce0ac130901301613i53be53bs2607c6fd86cc4222@mail.gmail.com>
	<3d375d730901301621l3391ea97id48ca8d674c257f4@mail.gmail.com>
Message-ID: <6ce0ac130901301631w74793303v1d8800a32a93e75a@mail.gmail.com>

>> I am using config.add_library to build a c++ library that I will link
>> into some Cython extensions.  This is working fine and generating a .a
>> library for me.  However, I need a shared library instead.  Is this
>> possible with numpy.distutils or will I need something like numscons?
>
> numscons or you can adapt the code from OOF2 and contribute it to
> numpy.distutils.

OK, thanks I hadn't seen the stuff in OOF2.

>> When calling add_library, what is the difference between the depends
>> and headers arguments?
>
> headers get installed via the distutils install_headers command. This
> will install the headers ... somewhere. Not exactly sure.

Again, thanks, that helps.

Cheers,

Brian

> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From cournape at gmail.com  Fri Jan 30 21:15:08 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 31 Jan 2009 11:15:08 +0900
Subject: [Numpy-discussion] NumPy distutils limitation?
In-Reply-To: <200901302130.59483.faltet@pytables.org>
References: <200901302130.59483.faltet@pytables.org>
Message-ID: <5b8d13220901301815p61a29089g59f21db5722bb7c0@mail.gmail.com>

On Sat, Jan 31, 2009 at 5:30 AM, Francesc Alted <faltet at pytables.org> wrote:
> Hi,
>
> Gregor and I are trying to give support for Intel's VML (Vector
> Mathematical Library) in numexpr.  For this, we are trying to make use
> of the weaponery in NumPy's distutils so as to be able to discover
> where the MKL (the package that contains VML) is located.  The
> libraries that we need to link with are: mkl_gf_lp64, mkl_gnu_thread,
> mkl_core *and* iomp5 (Intel's OpenMP library).
>
> The problem is that I have installed MKL as part as the Intel compiler
> for Unix.  In this setup, most of the libraries are in one place,
> namely:
>
> /opt/intel/Compiler/11.0/074/mkl/lib/em64t/
>
> However, the OpenMP library is in another directory:
>
> /opt/intel/Compiler/11.0/074/lib/intel64
>
> So, I need to specify *two* directories to get the complete set of
> libraries.  My first attempt was setting a site.cfg like:
>
> [DEFAULT]
> #libraries = gfortran
>
> [mkl]
> library_dirs= /opt/intel/Compiler/11.0/074/mkl/lib/em64t/:/opt/intel/Compiler/11.0/074/lib/intel64
> include_dirs =  /opt/intel/Compiler/11.0/074/mkl/include/
> mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core, iomp5
>
>
> Unfortunately, distutils complains and says that it cannot find the
> complete set of libraries:
>
> mkl_info:
>  libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found
> in /opt/intel/Compiler/11.0/074/mkl/lib/em64t/
>  libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found
> in /opt/intel/Compiler/11.0/074/lib/intel64
>  NOT AVAILABLE
>
> After some debugging of the problem, it seems that distutils needs to
> find *all* the required libraries in *one* single directory.

Yes

>
> My question is, is there an elegant way to handle this problem, or it
> is a limitation of the current distutils?

The "elegant" solution is to make softlink on unix.

> If the later, it would be
> nice it that could be solved in a future version, and several libraries
> can be found in *several* directories.

Unfortunately, this would mean rewriting system_info, because this
assumption is deeply ingrained in the whole design. Personally, I
think the really idea of looking for files for libraries is not the
right one - all other tools I know (autoconf, scons, jam, cmake) link
code snippet instead. But doing it in distutils won't be fun.

David


From gruben at bigpond.net.au  Fri Jan 30 19:00:20 2009
From: gruben at bigpond.net.au (Gary Ruben)
Date: Sat, 31 Jan 2009 11:00:20 +1100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
Message-ID: <49839494.9040104@bigpond.net.au>

The only time I've done this, I used numpy.fromfile exactly as follows. 
The file had a header followed by a number of records (one float 
followed by 128 complex numbers), requiring separate calls of 
numpy.fromfile to read each part. The only strange part about this was 
that the Fortran code was only supposed to be outputting 3 header fields 
but was adding an extra integer field for some unknown reason. I used a 
binary file viewer to work out the actual format of the file. To get at 
just the complex data in the records, I viewed the data as a recarray. 
Hopefully it's reasonably self-explanatory:

---

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm

f = open("field.dat",'rb')

# Read header
a = np.fromfile(f, np.dtype([('f1','<i4'), ('f2','<i4'), ('f3','<f8'),
                              ('f4','<i4')]), count = 1)

# Read data records
b = np.fromfile(f, np.dtype([('f1','<f8'), ('f2','128<c16')]))
bv = b.view(np.recarray)

plt.imshow(np.abs(np.fft.fftshift(np.fft.fft(bv.f2, axis=1), axes=[1])),
            origin='lower', interpolation='nearest', cmap=cm.Oranges)
plt.show()

---

Gary

David Froger wrote:
> Hy,
> 
> My question is about reading Fortran binary file (oh no this question 
> again...)
>
<snip>
> 
> After googling, I read that fopen and npfille was deprecated, and we 
> should use numpy.fromfile and ndarray.tofile, but despite of the 
> documentaion, the cookbook, the mailling list and google, I don't succed 
> in making a simple example. Considering the simple Fortran code below 
> what is the Python script to read the four arrrays? What about if my pc 
> is little endian and the file big endian?



From f.yw at hotmail.com  Fri Jan 30 23:41:57 2009
From: f.yw at hotmail.com (frank wang)
Date: Fri, 30 Jan 2009 21:41:57 -0700
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <3d375d730901301252j14628de9ye143b9d1aa3f18b5@mail.gmail.com>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>
	<3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
	<BLU105-W51137769905200294A75F789C60@phx.gbl> 
	<3d375d730901301252j14628de9ye143b9d1aa3f18b5@mail.gmail.com>
Message-ID: <BLU105-W19068EFC2818F1C465174189C70@phx.gbl>


Thanks for the correction. I will learn the ravel() function since I do not know it. Moving from Matlab world into python is tricky sometime.
 
Your output
In [22]: out> Out[22]: array([ 1.+3.j, -5.+9.j])> > In [23]: error> Out[23]: array([ 0.4-0.3j, 0.1-0.7j])
are correct answer.
 
However, if my data set is large, this solution takes long time to run. Are there any python/numpy magic to speed it up?
 
Thanks
 
Frank> Date: Fri, 30 Jan 2009 14:52:27 -0600> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] help on fast slicing on a grid> > On Fri, Jan 30, 2009 at 12:58, frank wang <f.yw at hotmail.com> wrote:> > I have created a test example for the question using for loop and hope> > someone can help me to get fast solution. My data set is about 2000000 data.> >> > However, I have the problem to run the code, the Out[i]=cnstl[j] line gives> > me error says:> >> > In [107]: Out[0]=cnstl[0]> > ---------------------------------------------------------------------------> > TypeError Traceback (most recent call last)> > C:\Frank_share\qamslicer.py in <module>()> > ----> 1> > 2> > 3> > 4> > 5> > TypeError: can't convert complex to float; use abs(z)> > In [108]: cnstl.dtype> > Out[108]: dtype('complex128')> >> > I do not know why that my data is complex128 already. Can anyone help to> > figure why?> > It's an odd error message, certainly. The root of the problem is that> you are attempting to put a (1,)-shaped array into a scalar. You don't> want to do that.> > By the way, you don't want to use the variable name Out in IPython.> It's already used to capture the output.> > > Thanks> >> > Frank> >> > from numpy import *> > a = arange(-15,16,2)> > cnstl=a.reshape(16,1)+1j*a> > cnstl=cnstl.reshape(256,1)> > Change that line to> > cnstl = cnstl.ravel()> > > X = array([1.4 + 1j*2.7, -4.9 + 1j*8.3])> > Out = array(X)> > error =array(X)> > for i in xrange(2):> > for j in xrange(256):> > a0 = real(X[i]) < (real(cnstl[j])+1)> > a1 = real(X[i]) > (real(cnstl[j])-1)> > a2 = imag(X[i]) > (imag(cnstl[j])-1)> > a3 = imag(X[i]) < (imag(cnstl[j])+1)> > if (a0 & a1 & a2 &a3):> > Out[i] = cnstl[j]> > error[i] = X[i] - cnstl[j]> > After reindenting this correctly, I get the following results:> > In [22]: out> Out[22]: array([ 1.+3.j, -5.+9.j])> > In [23]: error> Out[23]: array([ 0.4-0.3j, 0.1-0.7j])> > > Are those correct?> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion
_________________________________________________________________
Windows Live? Hotmail?:?more than just e-mail. 
http://windowslive.com/explore?ocid=TXT_TAGLM_WL_t2_hm_justgotbetter_explore_012009
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090130/5734aa7a/attachment.html>

From robert.kern at gmail.com  Sat Jan 31 00:08:35 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 30 Jan 2009 23:08:35 -0600
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <BLU105-W19068EFC2818F1C465174189C70@phx.gbl>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>
	<3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
	<BLU105-W51137769905200294A75F789C60@phx.gbl>
	<3d375d730901301252j14628de9ye143b9d1aa3f18b5@mail.gmail.com>
	<BLU105-W19068EFC2818F1C465174189C70@phx.gbl>
Message-ID: <3d375d730901302108v6ebc3215jd1d7871c43af1e51@mail.gmail.com>

On Fri, Jan 30, 2009 at 22:41, frank wang <f.yw at hotmail.com> wrote:
> Thanks for the correction. I will learn the ravel() function since I do not
> know it. Moving from Matlab world into python is tricky sometime.
>
> Your output
> In [22]: out
>> Out[22]: array([ 1.+3.j, -5.+9.j])
>>
>> In [23]: error
>> Out[23]: array([ 0.4-0.3j, 0.1-0.7j])
>
> are correct answer.
>
> However, if my data set is large, this solution takes long time to run. Are
> there any python/numpy magic to speed it up?

from numpy import *

a = arange(-15,16,2)
cnstl = a[:,newaxis] + 1j*a
cnstl = cnstl.ravel()
X = array([1.4 + 1j*2.7, -3.9 + 1j*8.3])

out = around((X + 1+1j) / 2.0) * 2.0 - (1+1j)
error = X - out

print out
print error

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From f.yw at hotmail.com  Sat Jan 31 02:26:25 2009
From: f.yw at hotmail.com (frank wang)
Date: Sat, 31 Jan 2009 00:26:25 -0700
Subject: [Numpy-discussion] help on fast slicing on a grid
In-Reply-To: <3d375d730901302108v6ebc3215jd1d7871c43af1e51@mail.gmail.com>
References: <49811036.5030701@ar.media.kyoto-u.ac.jp>
	<c7009a550901281955v29eaac7dsaf30b3b42dca9cd8@mail.gmail.com>
	<BLU105-W4686A3A7003F1615BDF27789C90@phx.gbl>
	<3d375d730901282157l4797a93ex293330d5c7d057c0@mail.gmail.com>
	<BLU105-W5073BF9D96E66E7B5F4EF889C90@phx.gbl>
	<3d375d730901282215t169158ccvfc406bc30a9cdd9a@mail.gmail.com>
	<BLU105-W51137769905200294A75F789C60@phx.gbl>
	<3d375d730901301252j14628de9ye143b9d1aa3f18b5@mail.gmail.com>
	<BLU105-W19068EFC2818F1C465174189C70@phx.gbl> 
	<3d375d730901302108v6ebc3215jd1d7871c43af1e51@mail.gmail.com>
Message-ID: <BLU105-W1335EFCEF3B8147E0CCC1489C70@phx.gbl>


Hi, Bob,
 
Thanks. This solution works great. It really helps me a lot. 
 
Frank> Date: Fri, 30 Jan 2009 23:08:35 -0600> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] help on fast slicing on a grid> > On Fri, Jan 30, 2009 at 22:41, frank wang <f.yw at hotmail.com> wrote:> > Thanks for the correction. I will learn the ravel() function since I do not> > know it. Moving from Matlab world into python is tricky sometime.> >> > Your output> > In [22]: out> >> Out[22]: array([ 1.+3.j, -5.+9.j])> >>> >> In [23]: error> >> Out[23]: array([ 0.4-0.3j, 0.1-0.7j])> >> > are correct answer.> >> > However, if my data set is large, this solution takes long time to run. Are> > there any python/numpy magic to speed it up?> > from numpy import *> > a = arange(-15,16,2)> cnstl = a[:,newaxis] + 1j*a> cnstl = cnstl.ravel()> X = array([1.4 + 1j*2.7, -3.9 + 1j*8.3])> > out = around((X + 1+1j) / 2.0) * 2.0 - (1+1j)> error = X - out> > print out> print error> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion
_________________________________________________________________
Windows Live? Hotmail?:?more than just e-mail. 
http://windowslive.com/explore?ocid=TXT_TAGLM_WL_t2_hm_justgotbetter_explore_012009
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090131/4c690031/attachment.html>

From sebastian.walter at gmail.com  Sat Jan 31 11:30:58 2009
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Sat, 31 Jan 2009 17:30:58 +0100
Subject: [Numpy-discussion] using numpy functions on an array of objects
In-Reply-To: <3d375d730901301305r6eb3cf11p5f4e8a0641bbf65c@mail.gmail.com>
References: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>
	<49835272.1060704@noaa.gov>
	<3d375d730901301305r6eb3cf11p5f4e8a0641bbf65c@mail.gmail.com>
Message-ID: <ec9f80fa0901310830w535cb7a7qdb3d29926ca2e4fe@mail.gmail.com>

Wouldn't it be nice to have numpy a little more generic?
All that would be needed was a little check of the arguments.

If I do:
numpy.trace(4)
shouldn't numpy be smart enough to regard the 4 as a 1x1 array?
numpy.sin(4) works!

and if
x = my_class(4)

wouldn't it be nice if

numpy.trace(x)
would call
x.trace() ?

numpy.sin(my_class(4)) works!

Wouldn't it be nice if numpy worked a little more consistent.
Is this worth a ticket? Or am I missing something here?











On Fri, Jan 30, 2009 at 10:05 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Fri, Jan 30, 2009 at 13:18, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>> I think you want to subclass an ndarray here. It's a bit tricky to so,
>> but if you look in the wiki and these mailing list archives, you'll find
>> advise on how to do it.
>
> That still won't work. numpy.linalg.inv() simply does a particular
> algorithm on float and complex arrays and nothing else.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From Mike.Colonno at spacex.com  Sat Jan 31 14:14:28 2009
From: Mike.Colonno at spacex.com (Mike Colonno)
Date: Sat, 31 Jan 2009 11:14:28 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <657E769D35612D4CAB1FEAACB0A92BB503094E5F3C@MAIL2.spacex.corp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<789d27b10901300432q28226381x829575e8b1fa23fe@mail.gmail.com>
	<9cca9e840901300731y69ee6f49ra758ba4f8ccced69@mail.gmail.com>,
	<9cca9e840901311048q21afab1cv7351403f8c95f3af@mail.gmail.com>,
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3C@MAIL2.spacex.corp>
Message-ID: <657E769D35612D4CAB1FEAACB0A92BB503094E5F3F@MAIL2.spacex.corp>


   Maybe I declared success too soon... The build / install of numpy seems to work (using MS C++ / Intel Fortran), but upon trying to import numpy I get:

>>>from numpy import *
Traceback (most recent call last):
  File "C:\<string>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\numpy\__init__.py", line 130, in <module>
    import add_newdocs
  File "C:\Python26\Lib\site-packages\numpy\add_newdocs.py", line 9, in <module>
    from lib import add_newdoc
  File "C:\Python26\Lib\site-packages\numpy\lib\__init__.py", line 161, in <module>
    from polynomial import *
  File "C:\Python26\Lib\site-packages\numpy\lib\polynomial.py", line 18, in <module>
    from numpy.linalg import eigvals, lstsq
  File "C:\Python26\Lib\site-packages\numpy\linalg\__init__.py", line 47, in <module>
    from linalg import *
  File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 22, in <module>
    from numpy.linalg import lapack_lite
ImportError: DLL load failed: Invalid access to memory location.

   To make matters more interesting, a second attempt leads to something different:

>>> from numpy import *
Traceback (most recent call last):
  File "C:\<string>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\numpy\__init__.py", line 130, in <module>
    import add_newdocs
  File "C:\Python26\Lib\site-packages\numpy\add_newdocs.py", line 9, in <module>
    from lib import add_newdoc
  File "C:\Python26\Lib\site-packages\numpy\lib\__init__.py", line 161, in <module>
    from polynomial import *
  File "C:\Python26\Lib\site-packages\numpy\lib\polynomial.py", line 11, in <module>
    import numpy.core.numeric as NX
AttributeError: 'module' object has no attribute 'core'

   Let me know if anyone has any thoughts here.

   Thanks for the help,

   ~Mike C.


From jonovik at gmail.com  Sat Jan 31 12:59:47 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Sat, 31 Jan 2009 17:59:47 +0000 (UTC)
Subject: [Numpy-discussion] ndarray.resize method and reference counting
References: <6a17e9ee0901130348o4781525ck4fb88072b05d352f@mail.gmail.com>
	<9457e7c80901130524k17e6015dh83e40319957320cb@mail.gmail.com>
	<6a17e9ee0901130645n41fcbc4ja11796c63d342685@mail.gmail.com>
Message-ID: <loom.20090130T211143-359@post.gmane.org>

Scott Sinclair <scott.sinclair.za <at> gmail.com> writes:

> >>> import numpy as np
> >>> x = np.eye(3)
> >>> x
> array([[ 1.,  0.,  0.],
>        [ 0.,  1.,  0.],
>        [ 0.,  0.,  1.]])
> >>> x.resize((5,5))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: cannot resize an array that has been referenced or is referencing
> another array in this way.  Use the resize function

I'm having the same problem, and have resigned to using "x = resize(x, 
newshape)" rather than "x.resize(newshape)" as suggested in the error message. 
Anything else becomes almost impossible to debug. Basically, you must never 
step into a function that does x.resize().

Part of the reason is explained here:
http://article.gmane.org/gmane.comp.python.numeric.general/5461

In my experience, reference counting behaves differently between:
* non-interactive running and simple IPython "run" of scripts (works as 
intended)
* interactive entering of commands in IPython (entering "x" at prompt "In [n]" 
defines an alias _n)
* debugging with "run -d" in Python (symbols in inner scopes seem to get extra 
references to them)

See the example below (using IPython 0.8.1/Python 2.5.2/Numpy 1.1.0/Linux) for 
illustration.

Hope this helps,
Jon Olav

====
In [10]: cat test.py
import numpy, sys

x = numpy.zeros(3)
print sys.getrefcount(x)
x.resize(2)
x
print sys.getrefcount(x)
x.resize(1)

def test():
    y = numpy.zeros(3)
    print sys.getrefcount(y)
    y.resize(2)
    y
    print sys.getrefcount(y)
    y.resize(1)

test()


In [11]: run test.py
2
2
2
2
=====

Non-interactively, getrefcount() returns 2 on all four occasions. Entering the 
same commands interactively, the refcount of x increases by 4 on entering "x" 
at the prompt. Inside a function call there is no problem, though.

=====
In [2]: import numpy, sys
In [3]: x = numpy.zeros(3)
In [4]: print sys.getrefcount(x)
2
In [5]: x.resize(2)
In [6]: x
Out[6]: array([ 0.,  0.])
In [7]: print sys.getrefcount(x)
6
In [8]: x.resize(1)
---------------------------------------------------------------------------
<type 'exceptions.ValueError'>: cannot resize an array that has been referenced 
or is referencing another array in this way.  Use the resize function

In [9]: def test():
   ...:         y = numpy.zeros(3)
   ...:         print sys.getrefcount(y)
   ...:         y.resize(2)
   ...:         y
   ...:         print sys.getrefcount(y)
   ...:         y.resize(1)
   ...:
In [10]: test()
2
2
=====

When debugging, however, the x.resize() works fine, while the y.resize() inside 
the test() function fails. (A workaround is to step over the function call, 
using "n" rather than "s" in line 19.)

====
In [5]: run -d test.py
ipdb> s
1---> 1 import numpy, sys
----> 3 x = numpy.zeros(3)
----> 4 print sys.getrefcount(x)
2

----> 5 x.resize(2)
----> 6 x
----> 7 print sys.getrefcount(x)
2

----> 8 x.resize(1)
---> 10 def test():

---> 19 test()
ipdb> s
---> 10 def test():
---> 11     y = numpy.zeros(3)
---> 12     print sys.getrefcount(y)
3

---> 13     y.resize(2)
ValueError: 'cannot resize an array that has been referenced or is referencing
\nanother array in this way.  Use the resize function'
=====




From robert.kern at gmail.com  Sat Jan 31 18:24:52 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 31 Jan 2009 17:24:52 -0600
Subject: [Numpy-discussion] using numpy functions on an array of objects
In-Reply-To: <ec9f80fa0901310830w535cb7a7qdb3d29926ca2e4fe@mail.gmail.com>
References: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>
	<49835272.1060704@noaa.gov>
	<3d375d730901301305r6eb3cf11p5f4e8a0641bbf65c@mail.gmail.com>
	<ec9f80fa0901310830w535cb7a7qdb3d29926ca2e4fe@mail.gmail.com>
Message-ID: <3d375d730901311524l70a05603vd06bea11d30e5675@mail.gmail.com>

On Sat, Jan 31, 2009 at 10:30, Sebastian Walter
<sebastian.walter at gmail.com> wrote:
> Wouldn't it be nice to have numpy a little more generic?
> All that would be needed was a little check of the arguments.
>
> If I do:
> numpy.trace(4)
> shouldn't numpy be smart enough to regard the 4 as a 1x1 array?

Why? It's not a 1x1 array. It's a scalar. If you want a 1x1 array,
give it a 1x1 array.

> numpy.sin(4) works!

Yes, numpy.sin() operates on scalars in addition to arrays.

> and if
> x = my_class(4)
>
> wouldn't it be nice if
>
> numpy.trace(x)
> would call
> x.trace() ?
>
> numpy.sin(my_class(4)) works!
>
> Wouldn't it be nice if numpy worked a little more consistent.
> Is this worth a ticket? Or am I missing something here?

numpy.sin() is a ufunc. Unary ufuncs will call the method of the same
name on objects in an object array (or the scalar itself if given an
object scalar). For example:

In [8]: class MyClass(object):
   ...:     def __init__(self, x):
   ...:         self.x = x
   ...:     def __repr__(self):
   ...:         return 'MyClass(%r)' % (self.x,)
   ...:     def sin(self):
   ...:         return MyClass(self.x+1)
   ...:
   ...:

In [9]: sin(MyClass(4))
Out[9]: MyClass(5)

In [10]: sin([MyClass(4), MyClass(5)])
Out[10]: array([MyClass(5), MyClass(6)], dtype=object)


You'll notice that numpy.sin() does not try to call the list.sin()
method when given the list. It interprets it as an object array, and
calls the MyClass.sin() method on each of the elements.

numpy.trace() is not an unary ufunc. It's just a function that
operates on (N>=2)-D arrays. You simply couldn't apply the same rules
as numpy.sin(). Otherwise, it would try to call the .trace() method on
each of the objects in your container, and obviously you can't
implement trace that way.

Having numpy.trace(x) simply call x.trace() would not be making numpy
more consistent.

Now, that said, the implementation of numpy.trace(x, *args) is
actually simply asarray(x).trace(*args). That should probably be
asanyarray(x) in order to allow ndarray subclasses. But this only
works because ndarray.trace() already exists. Making every function in
numpy check for a method first is just not going to happen.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco