From Oliphant.Travis@mayo.edu Thu Sep 2 00:34:33 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Wed, 1 Sep 1999 18:34:33 -0500 (CDT)
Subject: [Matrix-SIG] Sparse matrices (again?)
Message-ID: <Pine.LNX.4.10.9909011813100.3326-100000@us2.mayo.edu>
Well, it's been a few months since I first talked about sparse matrices
and like many of you I've been busy cleaning up some other issues on my
plate (and taking a vacation). Well, now I really need sparse matrix
support so I'm spending some time on it.
There were some excellent comments back in May and quite a bit of interest
which is why I'm mailing this list a progress-report-of-sorts so that I
can avoid unnecessary duplication of effort if possible and so I can
benefit from the more knowledgeable people out there (and solicit any
help by interested people).
I've wrapped up most of sparsekit2 and some of the sparse blas (from NIST)
after making changes to them so they will support single and double
precision for real and complex matrices. These packages will provide the
core (fast) functionality of sparse matrix operations and conversions from
one format to another.
Sparse kit contains conversion routines to convert between 15 different
formats but only contains basic matrix operations for compressed sparse
row format.
The sparse blas contains matrix-vector operations for many different
formats also but I have only wrapped up compressed sparse row and
compressed sparse column (as these were the only ones I converted from
double-precision-only to all-precisions as a start).
Now I'm setting up the Python class which will call these underlying
routines to do the operations and this is where I need some advice.
I'd thought to name the class spMatrix and subclass it from the Array base
class (thanks to the ExtensionClass foundation for NumPy 1.12), but I
wonder what the benefit is of doing that since I will need to redefine
most operations anyway. Any ideas?
When I get a fleshed out sparse class I'll post what I've got.
Alternatively if someone as a server they where CVS development could take
place that they'd be willing to donate I would place it there.
Just letting those interested know that work is progressing.
Travis,
From Oliphant.Travis@mayo.edu Thu Sep 2 03:34:59 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Wed, 1 Sep 1999 21:34:59 -0500 (CDT)
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible
Message-ID: <Pine.LNX.4.10.9909012126540.3715-100000@us2.mayo.edu>
I just got bit by an area where NumPy 1.12 is not 100% compatible with
older versions when using it in C.
If you use the O! syntax in PyArg_ParseTuple to check for array object
arguments (by using &PyArray_Type as the extra argument) you will have
trouble with your extension module with the NumPy 1.12 release since the
object passed is now a subclass of the extension class.
I don't know of an easy fix other than using O& with a converter function
that uses PyArray_Check to check or going to the old
check-each-argument-manually approach.
If anyone knows of a better approach let me know.
Thanks
Travis
From da@ski.org Thu Sep 2 05:01:23 1999
From: da@ski.org (David Ascher)
Date: Wed, 1 Sep 1999 21:01:23 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100%
compatible
In-Reply-To: <Pine.LNX.4.10.9909012126540.3715-100000@us2.mayo.edu>
Message-ID: <Pine.WNT.4.05.9909012059330.173-100000@david.ski.org>
On Wed, 1 Sep 1999, Travis Oliphant wrote:
> I just got bit by an area where NumPy 1.12 is not 100% compatible with
> older versions when using it in C.
>
> If you use the O! syntax in PyArg_ParseTuple to check for array object
> arguments (by using &PyArray_Type as the extra argument) you will have
> trouble with your extension module with the NumPy 1.12 release since the
> object passed is now a subclass of the extension class.
>
> I don't know of an easy fix other than using O& with a converter function
> that uses PyArray_Check to check or going to the old
> check-each-argument-manually approach.
>
> If anyone knows of a better approach let me know.
Ouch. I can't think of a solution that doesn't involve patching the core.
If you write the converter function, let me know and I'll include it
somewhere relevant.
--david ascher
From pearu@ioc.ee Thu Sep 2 07:59:45 1999
From: pearu@ioc.ee (Pearu Peterson)
Date: Thu, 2 Sep 1999 09:59:45 +0300 (EETDST)
Subject: [Matrix-SIG] Sparse matrices (again?)
In-Reply-To: <Pine.LNX.4.10.9909011813100.3326-100000@us2.mayo.edu>
Message-ID: <Pine.HPX.4.05.9909020950270.11272-100000@egoist.ioc.ee>
On Wed, 1 Sep 1999, Travis Oliphant wrote:
>
> I've wrapped up most of sparsekit2 and some of the sparse blas (from NIST)
> after making changes to them so they will support single and double
> precision for real and complex matrices. These packages will provide the
> core (fast) functionality of sparse matrix operations and conversions from
> one format to another.
Are you using f2py.py for wrapping? I think that I should then
keep f2py.py project alive as people may already use it and its serves
raw interfacing idea perfectly (I am not sure that PyFort ever will, or
at least in near future).
Pearu
From hinsen@cnrs-orleans.fr Fri Sep 3 15:03:18 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Fri, 3 Sep 1999 16:03:18 +0200
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible
In-Reply-To: <Pine.LNX.4.10.9909012126540.3715-100000@us2.mayo.edu> (message
from Travis Oliphant on Wed, 1 Sep 1999 21:34:59 -0500 (CDT))
References: <Pine.LNX.4.10.9909012126540.3715-100000@us2.mayo.edu>
Message-ID: <199909031403.QAA31181@chinon.cnrs-orleans.fr>
> If you use the O! syntax in PyArg_ParseTuple to check for array object
> arguments (by using &PyArray_Type as the extra argument) you will have
> trouble with your extension module with the NumPy 1.12 release since the
> object passed is now a subclass of the extension class.
I'd prefer a fix in NumPy, if possible. Without looking at any code
at all, could one rename the type of the "fundamental" array object
to something else and use PyArray_Type for the subclass? Assuming
of course that subclasses have proper type objects.
> I don't know of an easy fix other than using O& with a converter function
> that uses PyArray_Check to check or going to the old
> check-each-argument-manually approach.
I wouldn't want to do either to my code, I have tons of such checks...
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From Oliphant.Travis@mayo.edu Fri Sep 3 17:18:50 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Fri, 3 Sep 1999 11:18:50 -0500 (CDT)
Subject: [Matrix-SIG] A sparse matrix implementation.
Message-ID: <Pine.LNX.4.10.9909031105510.9595-100000@us2.mayo.edu>
I've got a sparse matrix implementation working based on the SPARSKIT
module by Yousef Saad. It's available currently only through cvs. If you
are interested in helping with this project (and have the time) let me
know and I can get access to the cvs server for you or just send you a
copy of the current source code.
If you have a cvs client then run
export CVSROOT=:pserver:anonymous@koer.ioc.ee:/usr/local/cvsroot
(or use setenv CVSROOT ****** if you run csh)
cvs login # the password is guest
cvs checkout Sparse
Once you have a version checked out run
cvs update -Pd
to get the latest version.
(If you have full access then you can modify the files and commit your
changes using
cvs commit -m "Some comment"
)
(You can checkout multipack from the same server in the same way if you
want (that is quite large though.))
Organization:
It is a base Python class with a helper extension module to
perform rapid calculations. The helper module is a wrapping of most of
the SPARSKIT toolbox.
I have toyed with the idea of building a sparse matrix Extension class
which may theoretical improve the speed of some operations and allow
for fast and fancy operations, but I don't know if it is worth the effort
or not. I'm interested in feedback on this issue.
Currently the internal format is Compressed Sparse Row but the structure
is set up to allow multiple formats.
Supported so far are matrix addition and subtraction, initialization from
coordinate format, and multiplication of a sparse matrix by a dense
vector.
Since sparse matrices inherently store indices there are a couple of
C-index style versus Fortran-index style questions to work out. The
SPARSEKIT library is in Fortran and uses that style. Right now this is
also the internal format of the class.
Check it out if you are interested.
Thanks,
Travis
From da@ski.org Fri Sep 3 18:23:48 1999
From: da@ski.org (David Ascher)
Date: Fri, 3 Sep 1999 10:23:48 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100%
compatible
In-Reply-To: <199909031403.QAA31181@chinon.cnrs-orleans.fr>
Message-ID: <Pine.WNT.4.04.9909031015500.210-100000@rigoletto.ski.org>
> I'd prefer a fix in NumPy, if possible. Without looking at any code
> at all, could one rename the type of the "fundamental" array object
> to something else and use PyArray_Type for the subclass? Assuming
> of course that subclasses have proper type objects.
Hmm. The problem is that currently all of the internal (C) code creates
objects of the fundamental data type, and all the .py code creates objects
of a subclass of said type. I think having this duality is in fact a very
good thing, as it means we can add features to the standard data type
which Python users manipulate w/o messing w/ C code.
Whether the fundamental data type (currently PyArray_Type) or the subclass
(Array) is used for a 'type equality' test, some code will break,
depending on whether it gets its arguments from a 'core' function or a
python function. I think the proper check is to do a subclass test
instead, which is what PyArray_Check() does. Alas, "O!" is unable to do
that right now. I suppose we could slip a PyArg_ParseTuple() which did it
in arrayobject.h. Nasty hack, though. (the 'right' solution will have to
wait for subclassability of Python types in the Python core, which isn't
planned until 2.0).
Any other ideas?
--david
From hinsen@cnrs-orleans.fr Fri Sep 3 18:42:47 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Fri, 3 Sep 1999 19:42:47 +0200
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100%
compatible
In-Reply-To: <Pine.WNT.4.04.9909031015500.210-100000@rigoletto.ski.org>
(message from David Ascher on Fri, 3 Sep 1999 10:23:48 -0700 (Pacific
Daylight Time))
References: <Pine.WNT.4.04.9909031015500.210-100000@rigoletto.ski.org>
Message-ID: <199909031742.TAA19514@chinon.cnrs-orleans.fr>
> Hmm. The problem is that currently all of the internal (C) code creates
> objects of the fundamental data type, and all the .py code creates objects
> of a subclass of said type. I think having this duality is in fact a very
> good thing, as it means we can add features to the standard data type
> which Python users manipulate w/o messing w/ C code.
Meaning that there are two array types which behave differently,
depending on where the array was created? Sounds like a developer's
nightmare to me. One of the strong points of Python is being able
to do the same thing in Python or C without making any difference to
the outside world. I'd like to keep that for arrays as well.
Is there any reason why C modules can't return the subclass as well?
> python function. I think the proper check is to do a subclass test
> instead, which is what PyArray_Check() does. Alas, "O!" is unable to do
But comparing type objects is a common approach in C extension
modules; I wouldn't be surprised if there was some code out there that
doesn't use the type check macro but a direct comparison. I am not
saying that I support this kind of style, but it was never explicitly
declared unsafe, and used work perfectly well.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From da@ski.org Fri Sep 3 19:15:23 1999
From: da@ski.org (David Ascher)
Date: Fri, 3 Sep 1999 11:15:23 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100%
compatible
In-Reply-To: <199909031742.TAA19514@chinon.cnrs-orleans.fr>
Message-ID: <Pine.WNT.4.04.9909031040000.210-100000@rigoletto.ski.org>
On Fri, 3 Sep 1999, Konrad Hinsen wrote:
> > Hmm. The problem is that currently all of the internal (C) code creates
> > objects of the fundamental data type, and all the .py code creates objects
> > of a subclass of said type. I think having this duality is in fact a very
> > good thing, as it means we can add features to the standard data type
> > which Python users manipulate w/o messing w/ C code.
>
> Meaning that there are two array types which behave differently,
> depending on where the array was created? Sounds like a developer's
> nightmare to me. One of the strong points of Python is being able
> to do the same thing in Python or C without making any difference to
> the outside world. I'd like to keep that for arrays as well.
Understood.
> Is there any reason why C modules can't return the subclass as well?
I don't know. Given the way the CAPI stuff works, it might be nontrivial
(the subclass is defined at Python runtime, so the CAPI struct has to be
modified then). It's worth investigating.
Alternatively, we could eliminate the subclass -- this should return
things to a backwards-compatible state. Currently, all the subclass does
is provide a place for the 'setting' of the array w/ a mask array. It's
not enough currently to justify breaking backwards compatibility. It was
my hope however that we could expand on that feature to provide the
gather/scatter functionality in Python, which I believe would be a very
significant feature, which is hard to do in C.
Whether we make the C modules return an 'official' subclass or remove that
subclass altogether, we still have a situation where there is no way to
define a subclass the instances of which are usable in C modules, if a
type equality test is used. This means that any such subclass will have
to be 'cast' before any call to a C extension module is made. Ugh.
> But comparing type objects is a common approach in C extension
> modules; I wouldn't be surprised if there was some code out there that
> doesn't use the type check macro but a direct comparison. I am not
> saying that I support this kind of style, but it was never explicitly
> declared unsafe, and used work perfectly well.
Indeed. I agree it's a problem. I'm not sure how big of a problem it is.
--david
From Oliphant.Travis@mayo.edu Fri Sep 3 23:50:14 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Fri, 3 Sep 1999 17:50:14 -0500 (CDT)
Subject: [Matrix-SIG] New Behavior for NumPy scalars?
Message-ID: <Pine.LNX.4.10.9909031738390.12074-100000@us2.mayo.edu>
O.K. I thought I had just gotten used to the old way of having NumPy
return Python scalars whenever a routine ends up with a NumPy scalar (not
necessarily a good idea when trying to keep everything as Floats).
Now, it looks like the math operations on NumPy scalars (rank-0 arrays)
result in NumPy scalars but being upcast to what a Python scalar of the
appropriate type would be. I don't think this is what is wanted. If we
are going to keep NumPy scalars as NumPy scalars after math results (I'm
all for this as it would alleviate many of my upcasting problems) we
should keep them the same type.
Example:
say b = array(3.0,'f')
a = array(2.0,'f')
It used to be that b-a returned a Python Float with value 1.0
Now b-a returns a NumPy Array of rank 0, with value 1.0 but with type 'd'.
I don't think anyone likes this idea.
The problem is that the __sub__ method of Array returns an Array all of
the time but subtract method returns a Python object for rank-0 results.
My personal solution preference is to not use PyArray_Return for the
ufuncs and let users who really need a scalar do the indexing to get it
from the rank-0 array. Using PyArray_Return seems like trying to
second-guess the user and leads to these kinds of issues.
Anyone else having any fun with the new NumPy 1.12 beta release?
--Travis
From Oliphant.Travis@mayo.edu Sat Sep 4 00:22:25 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Fri, 3 Sep 1999 18:22:25 -0500 (CDT)
Subject: [Matrix-SIG] A fix to recover old behavior for 1.12beta
Message-ID: <Pine.LNX.4.10.9909031819490.12159-100000@us2.mayo.edu>
This fix recovers the old behavior for rank-0 arrays (i.e. operations
returning rank-0 arrays are converted to Python scalars---again, I don't
think I like this behavior, though).
Added:
def _arrayret(self,val,copy=0):
if isarray(val):
return self.__class__(val,copy)
else:
return val
to the Array class
and replaced self.__class__ with self._arrayret in the class.
--Travis
From Oliphant.Travis@mayo.edu Sat Sep 4 00:48:00 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Fri, 3 Sep 1999 18:48:00 -0500 (CDT)
Subject: [Matrix-SIG] Previous solution dumps core due to typo and long-standing bug.
Message-ID: <Pine.LNX.4.10.9909031837520.12254-100000@us2.mayo.edu>
The previous solution I just posted dumped core when used on arrays
with rank > 0 (I had just tried it with rank-0 arrays).
The mistake is that copy should be copy=copy in the call to self.__class__
in _arrayret.
The reason it dumps core is due to a long-standing bug that has still not
been fixed the solution to which was posted by Charles Waldman May 26.
The C-code uses the typecode argument as a string without first making
sure it is a string:
The following (indicated with +) needs to be added to both array_array in
multiarraymodule.c and now array_init in arraymodule.c:
if (tpo == Py_None) {
type = PyArray_NOTYPE;
} else {
+ if (!PyString_Check(tpo)){
+ PyErr_SetString(PyExc_TypeError,
+ "typecode must be a character");
+ return NULL;
+ }
tp = PyString_AsString(tpo);
if (tp[0] == 0) type = PyArray_NOTYPE;
else type = tp[0];
Travis
From frank@ned.dem.csiro.au Mon Sep 6 07:32:03 1999
From: frank@ned.dem.csiro.au (Frank Horowitz)
Date: Mon, 6 Sep 1999 14:32:03 +0800
Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy?
Message-ID: <v04210100b3f90ee0c22b@[130.116.145.101]>
G'Day folks,
After spending the last couple of days strategizing on how to
fool the RPM system and chasing unresolved symbols in the linker on
an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building
NumPy (the LinearAlgebra.py contents, at least) linked with the
optimized BLAS from the ASCI Red project.
Has anyone else made this work? What are the tricks you used? :-)
TIA for any help!
Frank Horowitz
--
Frank Horowitz frank@ned.dem.csiro.au
Australian Geodynamics Cooperative Research Centre, and
CSIRO-Exploration & Mining, PO Box 437, Nedlands, WA 6009, AUSTRALIA
Direct: +61 8 9284 8431; FAX: +61 8 9389 1906; Reception: +61 8 9389 8421
From hinsen@cnrs-orleans.fr Mon Sep 6 11:20:29 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Mon, 6 Sep 1999 12:20:29 +0200
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100%
compatible
In-Reply-To: <Pine.WNT.4.04.9909031040000.210-100000@rigoletto.ski.org>
(message from David Ascher on Fri, 3 Sep 1999 11:15:23 -0700 (Pacific
Daylight Time))
References: <Pine.WNT.4.04.9909031040000.210-100000@rigoletto.ski.org>
Message-ID: <199909061020.MAA01059@chinon.cnrs-orleans.fr>
> > Is there any reason why C modules can't return the subclass as well?
>
> I don't know. Given the way the CAPI stuff works, it might be nontrivial
> (the subclass is defined at Python runtime, so the CAPI struct has to be
> modified then). It's worth investigating.
The C module would have to import the Python module, and that sounds
like a nice case of circular imports... Not so simple indeed...
> Alternatively, we could eliminate the subclass -- this should return
> things to a backwards-compatible state. Currently, all the subclass does
That might be the best immediate solution, until we find a better one.
> Whether we make the C modules return an 'official' subclass or remove that
> subclass altogether, we still have a situation where there is no way to
> define a subclass the instances of which are usable in C modules, if a
> type equality test is used. This means that any such subclass will have
> to be 'cast' before any call to a C extension module is made. Ugh.
That should be a general problem with extension classes, and perhaps
one should look for a general solution. Ideally, PyArg_ParseTuple()
(or a similar function provided by the ExtensionClass module) would
offer both direct type comparison and subclass checks. Then of course
C modules would have to be adapted - but I'd prefer to impose this
only once, when a definite solution is available.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From hinsen@cnrs-orleans.fr Mon Sep 6 14:36:41 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Mon, 6 Sep 1999 15:36:41 +0200
Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy?
In-Reply-To: <v04210100b3f90ee0c22b@[130.116.145.101]> (message from Frank
Horowitz on Mon, 6 Sep 1999 14:32:03 +0800)
References: <v04210100b3f90ee0c22b@[130.116.145.101]>
Message-ID: <199909061336.PAA21108@chinon.cnrs-orleans.fr>
> fool the RPM system and chasing unresolved symbols in the linker on
> an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building
> NumPy (the LinearAlgebra.py contents, at least) linked with the
> optimized BLAS from the ASCI Red project.
I got it linked after some trying, under RedHat 5.2. But the results
were not always the same (sometimes very different) as I got from a
straight Fortran BLAS, so I dropped it.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From hinsen@cnrs-orleans.fr Mon Sep 6 14:37:08 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Mon, 6 Sep 1999 15:37:08 +0200
Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy?
In-Reply-To: <v04210100b3f90ee0c22b@[130.116.145.101]> (message from Frank
Horowitz on Mon, 6 Sep 1999 14:32:03 +0800)
References: <v04210100b3f90ee0c22b@[130.116.145.101]>
Message-ID: <199909061337.PAA21112@chinon.cnrs-orleans.fr>
> After spending the last couple of days strategizing on how to
> fool the RPM system and chasing unresolved symbols in the linker on
> an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building
> NumPy (the LinearAlgebra.py contents, at least) linked with the
> optimized BLAS from the ASCI Red project.
I got it linked after some trying, under RedHat 5.2. But the results
were not always the same (sometimes very different) as I got from a
straight Fortran BLAS, so I dropped it.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From hinsen@cnrs-orleans.fr Mon Sep 6 14:45:01 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Mon, 6 Sep 1999 15:45:01 +0200
Subject: [Matrix-SIG] New Behavior for NumPy scalars?
In-Reply-To: <Pine.LNX.4.10.9909031738390.12074-100000@us2.mayo.edu> (message
from Travis Oliphant on Fri, 3 Sep 1999 17:50:14 -0500 (CDT))
References: <Pine.LNX.4.10.9909031738390.12074-100000@us2.mayo.edu>
Message-ID: <199909061345.PAA21116@chinon.cnrs-orleans.fr>
> My personal solution preference is to not use PyArray_Return for the
> ufuncs and let users who really need a scalar do the indexing to get it
> from the rank-0 array. Using PyArray_Return seems like trying to
> second-guess the user and leads to these kinds of issues.
It does what a naive user would expect, and what works well for the
vast majority of applications. Anyone who thinks that returning
rank-0 arrays is a good idea should volunteer for writing the
chapter in the documentation which explains the difference between
scalars and rank-0 arrays and why it is necessary to have both!
If you want to introduce rank-0 arrays consistently, you also have
to return a rank-0 result from indexing, for example. And then code
like
a = Numeric.zeros((3,))
a[0] = 1
...
if a[0] == 1:
...
would do something very unexpected.
The best solution would be to integrate the basic array module with
the standard Python distribution and make all Python scalars rank-0
arrays. Perhaps this could be proposed seriously for Python 2.0.
> Anyone else having any fun with the new NumPy 1.12 beta release?
From what you report, it looks as if I could put my time to a better
use...
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From strang@NMR.MGH.Harvard.EDU Mon Sep 6 22:09:06 1999
From: strang@NMR.MGH.Harvard.EDU (Gary Strangman)
Date: Mon, 6 Sep 1999 17:09:06 -0400 (EDT)
Subject: [Matrix-SIG] Creative slicing
Message-ID: <Pine.GSO.3.96.990906170505.25217D-100000@bucky.nmr.mgh.harvard.edu>
Hi all,
I've recently come across a need that someone on this list might be able
to help with. I'm trying to do the following with good speed using numpy
...
x = lots_o_data
x.shape = (100,40,50)
y = indices_into_data
y.shape = (40,50) # filled with ints in the 0-99 range
I want z where z.shape=(40,50) and the elements of z are chosen from x
according to the corresponding index in y. Make sense? That is, if
y[0,0] = 21 then I want z[0,0] = x[21,0,0], and so on. take() doesn't
seem capable of this. Might indices() be combined with take() or
something like that? Maybe there's something hidden down deep inside
numpy that could do this for me. I'm looking for a shape-independent
solution (i.e., so I can make my function ufunc-like).
Or is this just wishful thinking ??? ;-)
Gary Strangman
--------------------------------------------------------------
Gary Strangman, PhD | Neural Systems Group
Office: 617-724-0662 | Massachusetts General Hospital
Home: 401-453-3796 | 13th Street, Bldg 149, Room 9103
strang@nmr.mgh.harvard.edu | Charlestown, MA 02129
http://www.nmr.mgh.harvard.edu/Neural_Systems_Group/gary/
From Oliphant.Travis@mayo.edu Mon Sep 6 22:21:09 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Mon, 6 Sep 1999 16:21:09 -0500 (CDT)
Subject: [Matrix-SIG] New Behavior for NumPy scalars?
In-Reply-To: <199909061345.PAA21116@chinon.cnrs-orleans.fr>
Message-ID: <Pine.LNX.4.10.9909061604360.20028-100000@us2.mayo.edu>
> > My personal solution preference is to not use PyArray_Return for the
> > ufuncs and let users who really need a scalar do the indexing to get it
> > from the rank-0 array. Using PyArray_Return seems like trying to
> > second-guess the user and leads to these kinds of issues.
>
> It does what a naive user would expect,
>
Perhaps, perhaps, not. When I was new to Python I remember the thing that
kept getting me was the difference between lists and multiarrays. This
caused a couple of minor hangups until I finally got it in my head that
Python had multiple objects and not just one (like the MATLAB I was used
to.) Once somebody realizes that it's not such a big leap to have Python
scalars and Numeric scalars.
> and what works well for the vast majority of applications.
Hmm. Unless you are trying to actually use the single precision
datatypes. Then, you have to be extremely careful not to lose precision
because your NumPy arrays are always getting converted to Python scalars.
If operations with NumPy arrays always returned NumPy arrays the precision
upcasting problem wouldn't really be the problem it is, one would just
have to define all of there constant scalars. Not a bad programming
practice at that.
> Anyone who thinks that returning
> rank-0 arrays is a good idea should volunteer for writing the
> chapter in the documentation which explains the difference between
> scalars and rank-0 arrays and why it is necessary to have both!
I'd do it. I think the reason is the same as why is it necessary to have
both lists and NumPy arrays for the beginning user.
> If you want to introduce rank-0 arrays consistently, you also have
> to return a rank-0 result from indexing, for example. And then code
> like
Yes, probably true.
>
> a = Numeric.zeros((3,))
> a[0] = 1
> ...
> if a[0] == 1:
> ...
> would do something very unexpected.
>
This would be confusing for beginners it is agreed. But there are always
confusing issues when learning a new system. Changing something now would
undoubtedly break all kinds of old code.
Still, it would be nice if rank-0 arrays could exist less ephemerally.
I know let's introduce a global flag.... :-)
> The best solution would be to integrate the basic array module with
> the standard Python distribution and make all Python scalars rank-0
> arrays. Perhaps this could be proposed seriously for Python 2.0.
I agree this would be the best solution.
Now, either way you have pain under some circumstances. The
question is which pain is more tolerable. Lately, for me the precision
pain has been, well, a pain.
Travis
From da@ski.org Mon Sep 6 23:28:34 1999
From: da@ski.org (David Ascher)
Date: Mon, 6 Sep 1999 15:28:34 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100%
compatible
In-Reply-To: <199909061020.MAA01059@chinon.cnrs-orleans.fr>
Message-ID: <Pine.WNT.4.05.9909061525100.168-100000@david.ski.org>
> [konrad]
> That should be a general problem with extension classes, and perhaps
> one should look for a general solution. Ideally, PyArg_ParseTuple()
> (or a similar function provided by the ExtensionClass module) would
> offer both direct type comparison and subclass checks. Then of course
> C modules would have to be adapted - but I'd prefer to impose this
> only once, when a definite solution is available.
I like that idea. We could write a PyArg_ParseTupleExt (or some other
more useful name) which, if it finds an O= with a corresponding argument
which in an EC, does the class equality (not the type equality, btw, since
that is true for all EC's!), and e.g. when it sees a O<, does the
subclassing test as well.
I'll send a summary email to Guido and Jim Fulton, and pick their brains
on the topic.
--david
From da@ski.org Mon Sep 6 23:44:51 1999
From: da@ski.org (David Ascher)
Date: Mon, 6 Sep 1999 15:44:51 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] PyArg_ParseTuple & ExtensionClasses
Message-ID: <Pine.WNT.4.05.9909061529580.168-100000@david.ski.org>
Release 12beta of NumPy makes NumPy arrays ExtensionClasses. As an
experiment, the 'base class' (written in C) is subclassed in the
Numeric.py file, which is what most users use to create arrays in Python.
Extension modules create arrays from the C modules, hence get the 'base
class'. It's been interesting seeing all the implications of moving from
a type to a class, given legacy code.
Problem: While PyArray_Check() was rewritten to do an ExtensionClass-aware
subclassing check (whereas any instance of the array type/class or of a
subclass thereof is considered an array for the purpose of the check),
there is no way to plug that check into the PyArg_ParseTuple() behavior
when the "O!" identifier is sent, along with the PyArray_Type as the
argument.
The short-term solution proposed by Travis Oliphant is to use O& and to
supply a converter function which does that checking.
A long-term solution proposed by Konrad Hinsen would be to have a new
version of PyArg_ParseTuple which accepts two new typecodes, where one
does 'isinstance' checking and the other does
'is-instance-of-a-subclass-of' checking. Off the top of my head, "O=" and
"O<" could be used for identifying those two new checks. The new function
could be called PyArg_ParseTupleEC or any other name.
The reason I'm writing you two is that 1) it's an ExtensionClass-generic
issue, and if there are other ways of doing PyArg_ParseTuple checks with
EC's, we'd love to hear about it, and 2) when Python merges types and
classes, these issues will crop up, so I thought Guido should hear about
it.
Note that O! does type equality checking and will return true for any
ExtensionClass object, not just PyArray_Type.
--david
From hinsen@cnrs-orleans.fr Tue Sep 7 11:05:31 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Tue, 7 Sep 1999 12:05:31 +0200
Subject: [Matrix-SIG] New Behavior for NumPy scalars?
In-Reply-To: <Pine.LNX.4.10.9909061604360.20028-100000@us2.mayo.edu> (message
from Travis Oliphant on Mon, 6 Sep 1999 16:21:09 -0500 (CDT))
References: <Pine.LNX.4.10.9909061604360.20028-100000@us2.mayo.edu>
Message-ID: <199909071005.MAA00519@chinon.cnrs-orleans.fr>
> > and what works well for the vast majority of applications.
>
> Hmm. Unless you are trying to actually use the single precision
> datatypes. Then, you have to be extremely careful not to lose precision
> because your NumPy arrays are always getting converted to Python scalars.
Right. And I wouldn't mind disabling the conversion to scalars for
single-precision rank-0 arrays. After all, there are no single-precision
scalars, so there's a good reason for not doing that conversion.
And beginners are unlikely to fall into this trap.
Does this sound like a useful compromise?
> I'd do it. I think the reason is the same as why is it necessary to have
> both lists and NumPy arrays for the beginning user.
But that's easy, they have different properties. Scalars and rank-0
arrays, at least those that correspond to Python scalar types, are
expected to behave absolutely identically.
> > a = Numeric.zeros((3,))
> > a[0] = 1
> > ...
> > if a[0] == 1:
> > ...
> > would do something very unexpected.
> >
>
> This would be confusing for beginners it is agreed. But there are always
And not only that: the correct version of the test in the example above
looks awful. And a test that would work for any sequence object a would
become somewhere between incredibly messy and impossible.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From Oliphant.Travis@mayo.edu Tue Sep 7 15:53:24 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Tue, 7 Sep 1999 09:53:24 -0500 (CDT)
Subject: [Matrix-SIG] New Behavior for NumPy scalars?
In-Reply-To: <199909071005.MAA00519@chinon.cnrs-orleans.fr>
Message-ID: <Pine.LNX.4.10.9909070950520.24232-100000@us2.mayo.edu>
> >
> > Hmm. Unless you are trying to actually use the single precision
> > datatypes. Then, you have to be extremely careful not to lose precision
> > because your NumPy arrays are always getting converted to Python scalars.
>
> Right. And I wouldn't mind disabling the conversion to scalars for
> single-precision rank-0 arrays. After all, there are no single-precision
> scalars, so there's a good reason for not doing that conversion.
> And beginners are unlikely to fall into this trap.
>
> Does this sound like a useful compromise?
Yes, I think that would be quite a useful compromise. Good idea.
>
> And not only that: the correct version of the test in the example above
> looks awful. And a test that would work for any sequence object a would
> become somewhere between incredibly messy and impossible.
True. Thanks Konrad. It's always good to get advice from the voice of
reason before implementing any impulsive born-of-frustration design
decision...
Travis
From Oliphant.Travis@mayo.edu Tue Sep 7 16:49:19 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Tue, 7 Sep 1999 10:49:19 -0500 (CDT)
Subject: [Matrix-SIG] A Patch to only convert rank-0 arrays if they match Python Objects
Message-ID: <Pine.LNX.4.10.9909071045280.24766-100000@us2.mayo.edu>
Following up on the compromise suggested by Konrad for conversion of
rank-0 arrays, I propose the following minor patch so that single
Aprecision rank-0 arrays do not get automatically converted to Python
scalars.
*** LLNLDistribution12beta.orig/Numerical/Src/arrayobject.c Mon Aug 9 13:36:24 1999
--- LLNLDistribution12beta/Numerical/Src/arrayobject.c Tue Sep 7 10:42:51 1999
***************
*** 443,449 ****
if (mp != NULL) Py_DECREF(mp);
return NULL;
}
! if (mp->nd == 0) {
op = array_item(mp, 0);
Py_DECREF(mp);
return op;
--- 443,449 ----
if (mp != NULL) Py_DECREF(mp);
return NULL;
}
! if (mp->nd == 0 && (mp->type_num==PyArray_LONG || mp->type_num==PyArray_DOUBLE || mp->type_num==PyArray_CDOUBLE || mp->type_num==PyArray_OBJECT) ) {
op = array_item(mp, 0);
Py_DECREF(mp);
return op;
-Travis
From Oliphant.Travis@mayo.edu Tue Sep 7 17:03:35 1999
From: Oliphant.Travis@mayo.edu (Travis Oliphant)
Date: Tue, 7 Sep 1999 11:03:35 -0500 (CDT)
Subject: [Matrix-SIG] (no subject)
Message-ID: <Pine.LNX.4.10.9909071059480.24867-100000@us2.mayo.edu>
There is a function called array_set in the arrayfnsmodule of the gist
package that may help you accomplish your goal.
I have not used it much but here is the calling syntax.
array_set(vals1, indices, vals2)
it operates like
vals1 [indices] = vals2
It may only work with 1-D arrays (i.e. flattened arrays) though. I'm not
sure.
This is a common operation and it would be a nice addition to Numeric.
I've worked around it most of the time by changing the way I think about
assinging and using the where command with the logical operaters.
Travis
From amullhau@zen-pharaohs.com Tue Sep 7 20:15:20 1999
From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt)
Date: Tue, 7 Sep 1999 15:15:20 -0400
Subject: [Matrix-SIG] Re: [Matrix-SIG]
References: <Pine.LNX.4.10.9909071059480.24867-100000@us2.mayo.edu>
Message-ID: <042001bef965$5a0f0720$a602550a@amullhau>
> vals1 [indices] = vals2
>
> This is a common operation and it would be a nice addition to Numeric.
It is an essential addition to Numeric.
Later,
Andrew Mullhaupt
From se6y095@public.uni-hamburg.de Wed Sep 8 09:31:24 1999
From: se6y095@public.uni-hamburg.de (Berthold Höllmann)
Date: 08 Sep 1999 10:31:24 +0200
Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy?
In-Reply-To: Frank Horowitz's message of "Mon, 6 Sep 1999 14:32:03 +0800"
References: <v04210100b3f90ee0c22b@[130.116.145.101]>
Message-ID: <m23dwpvm77.fsf@pchoel.privat.uni-hamburg.de>
Frank Horowitz <frank@ned.dem.csiro.au> writes:
> G'Day folks,
>
> After spending the last couple of days strategizing on how to
> fool the RPM system and chasing unresolved symbols in the linker on
> an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building
> NumPy (the LinearAlgebra.py contents, at least) linked with the
> optimized BLAS from the ASCI Red project.
>
> Has anyone else made this work? What are the tricks you used? :-)
>
The line
lapack_lite -I./Include Src/lapack_litemodule.c -llapack -lblas -lg2c -DEXTENSIONCLASS -DCOMPILING_NUMPY
in Numerical/Setup.in did it for me. I linked lsblasppro1.1o_08.99.a
to libblas.a and lapack-Linux-i686.a to liblapack.a, lapack beeing
LAPACK 3.0. Note that this is from using LLNLDistribution12beta and
gcc 2.95.1.
Greetings
Berthold
--
bhoel@starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
From da@ski.org Thu Sep 9 01:11:58 1999
From: da@ski.org (David Ascher)
Date: Wed, 8 Sep 1999 17:11:58 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] New bug reporting system for NumPy
Message-ID: <Pine.WNT.4.04.9909081708220.241-100000@rigoletto.ski.org>
With gracious help from Greg Stein, we now have an online bug reporting
system for NumPy. We welcome additions to the bug database from all
users.
The URL for the system is: http://www.pythonpros.com/NumPy/bugs/
Please login (this creates an account for you to use in the future), and
enter all the bug reports you wish to. Please view the existing problem
reports to avoid entering duplicates. We are in the process of entering
previously submitted bug reports, but if you submitted a bug report in the
past which isn't already in the database, feel free to add it yourself.
Questions and feedback about this system should go to
support@icf.llnl.gov.
Thanks for your help,
--David Ascher & Paul Dubois
From jhauser@ifm.uni-kiel.de Thu Sep 9 08:11:19 1999
From: jhauser@ifm.uni-kiel.de (Janko Hauser)
Date: Thu, 9 Sep 1999 09:11:19 +0200 (CEST)
Subject: [Matrix-SIG] Creative slicing
In-Reply-To: <Pine.GSO.3.96.990906170505.25217D-100000@bucky.nmr.mgh.harvard.edu>
References: <Pine.GSO.3.96.990906170505.25217D-100000@bucky.nmr.mgh.harvard.edu>
Message-ID: <14295.23959.889135.630357@ifm.uni-kiel.de>
Try this code. It is not general for the number of dimensions, but
general for the length of the dimensions. Perhaps it is a start...
HTH,
__Janko
####### cut here
from Numeric import *
x=arange(10*4*5)
x.shape = (10,4,5)
y = ones((4,5))*2
y[0,0]=3
y[0,1]=4
y[1,0]=5
y.shape = (4,5) # filled with ints in the 0-9 range
# These lines need to be adapted for more dimensions
yi,xi=indices(y.shape)
linindex = y*product(x.shape[1:])+yi*y.shape[-2]+xi
z=take(ravel(x),ravel(linindex))
z.shape=(4,5)
From Gary Strangman <strang@NMR.MGH.Harvard.EDU> Fri Sep 10 00:35:06 1999
From: Gary Strangman <strang@NMR.MGH.Harvard.EDU> (Gary Strangman)
Date: Thu, 9 Sep 1999 19:35:06 -0400 (EDT)
Subject: [Matrix-SIG] Creative slicing
In-Reply-To: <14295.23959.889135.630357@ifm.uni-kiel.de>
Message-ID: <Pine.GSO.3.96.990909172206.27621B-100000@bucky.nmr.mgh.harvard.edu>
> Try this code. It is not general for the number of dimensions, but
> general for the length of the dimensions. Perhaps it is a start...
<snip>
A good start indeed. Thanks! I just thought I'd pass along the following
code to any others who might be interested. (In the FWIW category, I
agree that this kind of slicing would be an excellent addition to the
NumPy core.)
> x=arange(10*4*5)
> x.shape = (10,4,5)
> y = ones((4,5))*2
> y[0,0]=3
> y[0,1]=4
> y[1,0]=5
> y.shape = (4,5) # filled with ints in the 0-9 range
>
> # These lines need to be adapted for more dimensions
> yi,xi=indices(y.shape)
> linindex = y*product(x.shape[1:])+yi*y.shape[-2]+xi
This wasn't quite right ... but very close (an off-by-one error, probably
a mail typo). Instead use ...
linindex = y*(product(x.shape[1:]))+yi*y.shape[1]+xi
> z=take(ravel(x),ravel(linindex))
> z.shape=(4,5)
Starting from your idea, I modified the code to work in the N-D case.
It's not exactly pretty (and I didn't bother to find a way around the
for-loop) but it seems to work. This general type of workaround is not
going to be fast for large arrays ... which is what would make it nice to
have such a function ... a[indices] = vals ... in the numpy core. ;-)
Gary
<------CUT BELOW------>
def elementslice(a, indexarray, dimension=0):
"""
Usage: elementslice(a, indexarray, dimension=0)
Returns: a[indices] along 'dimension', resulting shape = indices.shape
"""
if dimension <> 0: # for convenience, put operating dimension first
newdims = range(len(a.shape))
del newdims[dimension]
newdims = [dimension] + newdims
a = N.transpose(a,newdims)
else:
newdims = a.shape
ind = N.indices(indexarray.shape)
linindex = indexarray*N.product(a.shape[1:]) # get offsets for 1st dim
for i in range(len(ind)-1): # for remaining dims but one, add in offsets
multindices = N.product(indexarray.shape[i+1:])
linindex = linindex + ind[i]*multindices
linindex = linindex + ind[-1] # add in last dim's offsets
slice = N.take(N.ravel(a),N.ravel(linindex)) # get vals from ravel'd array
return N.reshape(slice,indexarray.shape) # reshape properly
--------------------------------------------------------------
Gary Strangman, PhD | Neural Systems Group
Office: 617-724-0662 | Massachusetts General Hospital
Home: xxx-xxx-xxxx | 13th Street, Bldg 149, Room 9103
strang at nmr.mgh.harvard.edu | Charlestown, MA 02129
http://www.nmr.mgh.harvard.edu/Neural_Systems_Group/gary/
From dubois1@llnl.gov Fri Sep 10 16:33:17 1999
From: dubois1@llnl.gov (Paul F. Dubois)
Date: Fri, 10 Sep 1999 08:33:17 -0700
Subject: [Matrix-SIG] Pyfort Version 2 now available
Message-ID: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com>
Version 2 of Pyfort, a tool that lets you extend Python with Fortran, is now
available. See http://xfiles.llnl.gov/python.htm.
To use Pyfort, you construct an input file in a Fortran-90 like syntax. This
input file contains interfaces for the functions in Fortran that are to be
called. Pyfort then constructs a C-language extension module that contains
the "glue" to turn Python calls into the calls to your Fortran.
Example:
Fortran:
function sum (n, x) ! return sum of elements of x
integer n
real x(n)
real sum
....implementation...
end function sum
Pyfort input:
module modsum
function sum (n, x) ! return sum of elements of x
integer n = size(x)
real x(n)
real sum
end function sum
end module modsum
Calling sum in Python:
from modsum import sum
x = [1., 2., 3.] # or a Numeric array
print sum(x)
print sum.__doc__ #prints "return sum of elements of x"
pyfort -c compiler modsum.pyf produces modsummodule.c and modsum.txt; the
former is used to construct the Python extension and the latter documents
the Python calling sequence. If no comment is given, the doc string is the
calling sequence.
Users can declare output quantities by using the intent attribute:
real, intent(out):: y(n)
Such an argument becomes an output of the function. If there are multiple
return values, a tuple is returned.
Intent "temporary" requests that an array be constructed and passed to the
Fortran as a work array.
Persons wanting a "raw" interface that is exactly like the Fortran routine
and which does as little as possible can achieve this by declaring any array
arguments or scalar outputs as intent(inout).
New for version 2:
Module attributes NONE, TRANSPOSE (default), MIRROR control behavior
of multiple dimension intent(in) and intent(output) arrays.
NONE -- pass argument as is; return array has transposed strides
to "look right" in Python.
TRANSPOSE -- transpose the data area on input; return array has
transposed strides. A returned array subsequently used in another call as
input will not in fact ever be transposed.
MIRROR -- the Fortran has been written with all the dimensions
reversed; check shapes backward and create output arrays with a backward
shape. No data movement ever required.
Method set_pyfort_option available to set the default option.
Each generated wrapper can take an optional extra argument of one of
these three attributes to override the default.
Scalar input arguments can be given an initial value as an expression
involving constants, the Fortran 90 size operator,
other scalar input arguments that were not given an initial value, and
scalar input arguments that occur earlier in the argument list. See argument
"n" above, for example. The size operator can be size(a) or size(a,
dimensionnumber) (Fortran style, 1-based).
Example: subroutine doit (n, m, x, k, y, z)
integer n = size(x, 1), m = size(x, 2) , k = n * m
real x(n, m)
real, intent(out):: y(k)
complex, intent(temporary):: z(n, m)
Python call: y = doit(x)
The purpose of this facility is to enable the construction of "nice"
interfaces without needing an additional Python layer in many cases.
Pyfort has been tested under Linux using the Portland Group (-c pgf77) and
g77 (-c g77) compilers, under Solaris (choose -c solaris) and SGI (-c
solaris works). Potential users should note that both a Fortran and C
compiler are needed. Pyfort has not yet been heavily used and bugs are
possible.
Documentation is in the file pyfort.pdf that comes with the package.
Pyfort has been released for free redistribution. See Legal.htm.
Questions to support@pcmdi.llnl.gov
Paul F. Dubois
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
<P><A HREF="http://xfiles.llnl.gov/python.htm">Pyfort 2</A> -
Python / Fortran connection tool. (9 - Sep - 99)
From dubois1@llnl.gov Fri Sep 10 18:12:20 1999
From: dubois1@llnl.gov (Paul F. Dubois)
Date: Fri, 10 Sep 1999 10:12:20 -0700
Subject: [Matrix-SIG] Numerical plans
Message-ID: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com>
This is a multi-part message in MIME format.
------=_NextPart_000_0007_01BEFB74.F8142340
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
After some consideration, David and I have decided to back out the =
changes related to Extension Class. We intend to release a version 12 =
that has the bug fixes since version 11 but not the Extension Class =
work.=20
Practical experience has given us a different idea for solving the =
problem of wanting to inherit from an array class. We want to explore =
this idea for a while. While the extension class approach basically =
works, it doesn't really turn out to ease the developer's task and it =
does raise a number of incompatibility issues that we hadn't forseen.=20
Older and wiser, we will try again.
Periodically the issue is raised of=20
x[indices] =3D values=20
x[indices] as an r-value
being desireable, as it was recently. As I pointed out before, everyone =
agrees that the above line is desireable, but when x has rank > 1, there =
are strongly differing views of what this ought to mean. Multiple =
intelligent, experienced developers have told me that (a) I am an idiot, =
because (b) there is only one obvious and highly desireable solution, =
namely theirs Unfortunately the latter does not turn out to be the same =
in all cases.
The issue is complicated by the fact that the coding for setindex is =
already totally incomprehensible and so changing it requires a great =
deal of care and testing.
It is time for proposals. Concrete, detailed. Discussion to follow. If =
we can come to a consensus it would be nice.
Do not forget things like NewAxis, Ellipsis, and the colon. How do these =
things mix and match?
------=_NextPart_000_0007_01BEFB74.F8142340
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META content=3D"MSHTML 5.00.2614.3401" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT size=3D2>After some consideration, David and I have decided =
to back out=20
the changes related to Extension Class. We intend to release a version =
12 that=20
has the bug fixes since version 11 but not the Extension Class work.=20
</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>Practical experience has given us a different idea =
for solving=20
the problem of wanting to inherit from an array class. We want to =
explore this=20
idea for a while. While the extension class approach basically works, it =
doesn't=20
really turn out to ease the developer's task and it does raise a number =
of=20
incompatibility issues that we hadn't forseen. </FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>Older and wiser, we will try again.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>Periodically the issue is raised of </FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>x[indices] =3D values </FONT></DIV>
<DIV><FONT size=3D2>x[indices] as an r-value</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>being desireable, as it was recently. As I pointed =
out before,=20
everyone agrees that the above line is desireable, but when x has rank =
> 1,=20
there are strongly differing views of what this ought to mean. Multiple=20
intelligent, experienced developers have told me that (a) I am an idiot, =
because=20
(b) there is only one obvious and highly desireable solution, namely =
theirs=20
Unfortunately the latter does not turn out to be the same in all=20
cases.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>The issue is complicated by the fact that the coding =
for=20
setindex is already totally incomprehensible and so changing it requires =
a great=20
deal of care and testing.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>It is time for proposals. Concrete, detailed. =
Discussion to=20
follow. If we can come to a consensus it would be nice.</FONT></DIV>
<DIV><FONT size=3D2>Do not forget things like NewAxis, Ellipsis, and the =
colon.=20
How do these things mix and match?</FONT></DIV>
<DIV> </DIV>
<DIV> </DIV></BODY></HTML>
------=_NextPart_000_0007_01BEFB74.F8142340--
From godzilla@netmeg.net (Les Schaffer) Fri Sep 10 18:35:34 1999
From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer)
Date: Fri, 10 Sep 1999 13:35:34 -0400 (EDT)
Subject: [Matrix-SIG] Numerical plans
In-Reply-To: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com>
References: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com>
Message-ID: <14297.16742.754718.207527@gargle.gargle.HOWL>
Paul DuBois said:
> Older and wiser, we will try again.
since you seem to be in reflection mode, i was wondering if you could
explain to the dummies among us (me, myself and i) what was the
purpose of this attempt at using extension classes for NumPy?
I never caught on, from the discussions here, what the value was
supposed to be to the -- ahem -- average user.
--
____ Les Schaffer ___| --->> Engineering R&D <<---
Theoretical & Applied Mechanics | Designspring, Inc.
Center for Radiophysics & Space Research | http://www.designspring.com/
Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com
From amullhau@zen-pharaohs.com Fri Sep 10 22:26:41 1999
From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt)
Date: Fri, 10 Sep 1999 17:26:41 -0400
Subject: [Matrix-SIG] Numerical plans
References: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com>
Message-ID: <033301befbd6$e9cf4280$a602550a@amullhau>
> Periodically the issue is raised of
> x[indices] = values
> x[indices] as an r-value
> being desireable, as it was recently.
And it will be, until it's in there or another more attractive language
comes along.
> As I pointed out before, everyone agrees that the above line is
desireable, but when > x has rank > 1, there are strongly differing views of
what this ought to mean. Multiple > intelligent, experienced developers have
told me that (a) I am an idiot, because (b)
> there is only one obvious and highly desireable solution, namely theirs
Unfortunately > the latter does not turn out to be the same in all cases.
I think these developers don't have much experience. There are clearly
multiple useful meanings that this can have. Especially depending on what
range indices can have in your language.
The one that probably makes the most sense is to coerce x to a rank-1 array,
and index that, return the rank-1 result. A lot of languages do that, and
nobody is banging on them to change.
> The issue is complicated by the fact that the coding for setindex is
already totally
> incomprehensible and so changing it requires a great deal of care and
testing.
Time for a replacement. There are some methods for indexing which are both
simpler and can be substantially faster.
> It is time for proposals. Concrete, detailed. Discussion to follow. If we
can come to a > consensus it would be nice.
Steal the S style, except you probably can't get away with the special
treatment of nonpositive index values and sparse indexing by high rank
arrays.
The APL style is better thought out in terms of language design, but then
you need to add more functions for the stuff cyclic extension does for you,
and the code you have to write isn't as fast. Experience has shown that the
APL approach was probably too rigid.
Later,
Andrew Mullhaupt
From se6y095@public.uni-hamburg.de Sat Sep 11 12:59:39 1999
From: se6y095@public.uni-hamburg.de (Berthold Höllmann)
Date: 11 Sep 1999 13:59:39 +0200
Subject: [Matrix-SIG] Pyfort Version 2 now available
In-Reply-To: "Paul F. Dubois"'s message of "Fri, 10 Sep 1999 08:33:17 -0700"
References: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com>
Message-ID: <m2vh9hy7ys.fsf@pchoel.privat.uni-hamburg.de>
"Paul F. Dubois" <dubois1@llnl.gov> writes:
> Version 2 of Pyfort, a tool that lets you extend Python with Fortran, is now
> available. See http://xfiles.llnl.gov/python.htm.
Hello,
I downloaded and installed pyfort2. Checking the test subdirectory I
get:
# python testit.py
itimes [1 2 3] [2 3 4]
[ 2 6 12]
times [ 1. 2. 3.] [ 2. 3. 4.]
[ 2. 6. 12.]
bang 1.2 [0 1 2 3 4 5 6 7 8 9] [ 3.+0.j 4.+2.j]
Speicherzugriffsfehler
(Speicherzugriffsfehler is german for "segmentation fault")
So I started gdb:
# gdb python
GNU gdb 4.17.0.11 with Linux support
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
(gdb) run testit.py
Starting program: /usr/local/bin/python testit.py
itimes [1 2 3] [2 3 4]
[ 2 6 12]
times [ 1. 2. 3.] [ 2. 3. 4.]
[ 2. 6. 12.]
bang 1.2 [0 1 2 3 4 5 6 7 8 9] [ 3.+0.j 4.+2.j]
Program received signal SIGSEGV, Segmentation fault.
0x4013730b in do_array_in (rname=0x4013a0da "bang", vname=0x4013a0d8 "R",
v=0x80cdcd4, python_array_type=PyArray_FLOAT) at ./testpyfmodule.c:172
172 if (t->descr->type_num != python_array_type) {
(gdb) where
#0 0x4013730b in do_array_in (rname=0x4013a0da "bang", vname=0x4013a0d8 "R",
v=0x80cdcd4, python_array_type=PyArray_FLOAT) at ./testpyfmodule.c:172
#1 0x40138103 in testpyf_bang (unused=0x0, args=0x80d4f50) at ./testpyfmodule.c:562
#2 0x80539e0 in call_builtin (func=0x80cebf8, arg=0x80d4f50, kw=0x0)
at ceval.c:2359
#3 0x80538f8 in PyEval_CallObjectWithKeywords (func=0x80cebf8, arg=0x80d4f50,
kw=0x0) at ceval.c:2324
#4 0x8052ac5 in eval_code2 (co=0x80cd7e0, globals=0x80a34d8, locals=0x80a34d8,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0)
at ceval.c:1654
#5 0x8050dd2 in PyEval_EvalCode (co=0x80cd7e0, globals=0x80a34d8, locals=0x80a34d8)
at ceval.c:324
#6 0x805f8ae in run_node (n=0x80bd658, filename=0xbffff8a4 "testit.py",
globals=0x80a34d8, locals=0x80a34d8) at pythonrun.c:887
#7 0x805f868 in run_err_node (n=0x80bd658, filename=0xbffff8a4 "testit.py",
globals=0x80a34d8, locals=0x80a34d8) at pythonrun.c:872
#8 0x805f840 in PyRun_File (fp=0x80a1148, filename=0xbffff8a4 "testit.py",
start=257, globals=0x80a34d8, locals=0x80a34d8) at pythonrun.c:860
#9 0x805efce in PyRun_SimpleFile (fp=0x80a1148, filename=0xbffff8a4 "testit.py")
at pythonrun.c:570
#10 0x805ec61 in PyRun_AnyFile (fp=0x80a1148, filename=0xbffff8a4 "testit.py")
at pythonrun.c:451
#11 0x804f5eb in Py_Main (argc=2, argv=0xbffff724) at main.c:287
#12 0x804f0e8 in main (argc=2, argv=0xbffff724) at python.c:12
(gdb) print t->descr->type_num
Cannot access memory at address 0x3ff00034.
(gdb) print t->descr
$1 = (PyArray_Descr *) 0x3ff00000
(gdb) print t->descr->type
Cannot access memory at address 0x3ff00044.
(gdb) print t->descr->data
There is no member named data.
(gdb) print t->ob_type
$2 = (struct _typeobject *) 0x8094cc0
(gdb) print t->ob_type->tp_name
$3 = 0x808c3a0 "float"
(gdb)
I'm using LLNLDistribution12beta which seems to return Floats instead
of array(<value>, Float) from PyArray_ContiguousFromObject for single
values. (This is my understanding of the recent dicussion on
[Matrix-SIG])
Cheers
Berthold
--
bhoel@starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
From se6y095@public.uni-hamburg.de Sat Sep 11 14:32:16 1999
From: se6y095@public.uni-hamburg.de (Berthold Höllmann)
Date: 11 Sep 1999 15:32:16 +0200
Subject: [Matrix-SIG] Pyfort Version 2 now available
In-Reply-To: se6y095@public.uni-hamburg.de's message of "11 Sep 1999 13:59:39 +0200"
References: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com> <m2vh9hy7ys.fsf@pchoel.privat.uni-hamburg.de>
Message-ID: <m2d7vpwp3z.fsf@pchoel.privat.uni-hamburg.de>
se6y095@public.uni-hamburg.de (Berthold Höllmann) writes:
> "Paul F. Dubois" <dubois1@llnl.gov> writes:
>
> > Version 2 of Pyfort, a tool that lets you extend Python with Fortran, is now
> > available. See http://xfiles.llnl.gov/python.htm.
>
> Hello,
>
> I downloaded and installed pyfort2. Checking the test subdirectory I
> get:
...
> I'm using LLNLDistribution12beta which seems to return Floats instead
> of array(<value>, Float) from PyArray_ContiguousFromObject for single
> values. (This is my understanding of the recent dicussion on
> [Matrix-SIG])
Oops,
I reinstalled LLNLDistribution11 and pyfort 1 and still get the
segmentation fault. BTW, this is python 1.5.2.
Cheers
Berthold
--
bhoel@starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
From dubois1@llnl.gov Sat Sep 11 16:20:59 1999
From: dubois1@llnl.gov (Paul F. Dubois)
Date: Sat, 11 Sep 1999 08:20:59 -0700
Subject: [Matrix-SIG] Pyfort Version 2 now available
References: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com> <m2vh9hy7ys.fsf@pchoel.privat.uni-hamburg.de>
Message-ID: <000d01befc69$41259420$3c810b18@plstn1.sfba.home.com>
Thanks for the bug report. I had not tested Pyfort with the Version 12 beta.
As I mentioned yesterday, we are abandoning that approach and your note is a
good reminder for me to go remove the beta distribution. Everyone should
reinstall version 11 for now.
I also need to mention that if you use Pyfort with LLNL 11, you may want to
edit the place in generator.py that says
#include "Numeric/arrayboject.h"
to make the N lower case, before installation. In the future the
installation is going to Numeric instead of numeric, but 11 still does it
the old way.
From turner@blueskystudios.com Mon Sep 13 21:27:05 1999
From: turner@blueskystudios.com (John A. Turner)
Date: Mon, 13 Sep 1999 16:27:05 -0400 (EDT)
Subject: [Matrix-SIG] Pickling
Message-ID: <14301.24089.837561.510054@waterloo.blueskyprod.com>
There was a short thread on NumPy and Pickling back in June, started
by this msg:
http://www.python.org/pipermail/matrix-sig/1999-June/002909.html
with followups by Hinson and Ascher. Just wondering where this stands
at the moment.
Thanks,
--
John A. Turner Senior Research Associate
Blue Sky Studios, One South Rd, Harrison, NY 10528
http://www.blueskystudios.com/ (914) 381-8400
From da@ski.org Tue Sep 14 04:54:02 1999
From: da@ski.org (David Ascher)
Date: Mon, 13 Sep 1999 20:54:02 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] Pickling
In-Reply-To: <14301.24089.837561.510054@waterloo.blueskyprod.com>
Message-ID: <Pine.WNT.4.05.9909132053020.157-100000@david.ski.org>
> There was a short thread on NumPy and Pickling back in June, started
> by this msg:
>
> http://www.python.org/pipermail/matrix-sig/1999-June/002909.html
>
> with followups by Hinson and Ascher. Just wondering where this stands
> at the moment.
Release 12beta has fixed pickling code (which could be stolen to 'fix'
version 11). We haven't investigated high-performance pickling much
further, due to the unsatisfactory state of understanding of the buffer
interface and general lack of time.
--david ascher
From dubois1@llnl.gov Tue Sep 14 15:28:31 1999
From: dubois1@llnl.gov (Paul F. Dubois)
Date: Tue, 14 Sep 1999 07:28:31 -0700
Subject: [Matrix-SIG] Re: Numerical Python
References: <Pine.HPP.3.96.990914143239.15430A-100000@barbes>
Message-ID: <002301befebd$6b697520$3c810b18@plstn1.sfba.home.com>
Dear Matrix-Sigers who have Macs,
Mr. Nataf seeks a Mac version of NumPy 11, and advice about a choice of
graphics package.
Thanks,
Paul
----- Original Message -----
From: Frederic Nataf <nataf@cmapx.polytechnique.fr>
To: <support@icf.llnl.gov>
Sent: Tuesday, September 14, 1999 5:39 AM
Subject: Numerical Python
> Dear Sir,
>
> I have heard about Numerical Python by a colleague of mine who spent
> one year at
> LLNL. I am very interested in working with it. I work both on Unix
> workstations (gcc) and on Macintosh (CodeWarrior). I have seen on your web
> tutorial
> http://xfiles.llnl.gov/NumDoc4.html#18177
> that python is available on the Macintosh along with a not so uptodate
> version of your code. So, I have two questions:
>
> 1) is it difficult or a problem to have the last version work on a
> Macintosh
>
> 2) what about the graphic package. Can it work on a Macintosh?
>
>
> Sincerly,
>
> Frederic Nataf
>
> PS. How MacOsX will change the situation?
>
>
From da@ski.org Tue Sep 14 17:14:26 1999
From: da@ski.org (David Ascher)
Date: Tue, 14 Sep 1999 09:14:26 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] Numerical plans (fwd)
Message-ID: <Pine.WNT.4.04.9909140914110.229-100000@rigoletto.ski.org>
On Fri, 10 Sep 1999, Les Schaffer wrote:
> Paul DuBois said:
>
> > Older and wiser, we will try again.
>
> since you seem to be in reflection mode, i was wondering if you could
> explain to the dummies among us (me, myself and i) what was the
> purpose of this attempt at using extension classes for NumPy?
>
> I never caught on, from the discussions here, what the value was
> supposed to be to the -- ahem -- average user.
Well, since I think I was the main force behind that, I'll say that the
main idea was that I wanted to be able to continue work on the
functionality of array objects in Python, not in C. The C code is a mess,
and I felt it much easier to do things like the indexing that Paul was
talking about (called many things my many people) in Python.
ExtensionClasses was supposed to allow this sort of subclassing easily.
Other ideas included things like being able to specify (again, in Python)
different coercion behaviors. Etc.
The issues we hit were:
1) ExtensionClass has some bugs, which are hard to tackle, and the author
(Jim Fulton) is too busy to track them down.
2) ExtensionClass is hard to reconcile with the variety of ways NumPy is
compiled (the CAPI, static/dynamic linking, etc.)
3) Making a type a class simply isn't doable in a backwards compatible way
while keeping the advantages of a class.
Instead, what Paul and I are planning to do is to revert to the old type,
but to do UserArray *right*. The new features that I mention above would
then be implemented in new Python classes, which would use array objects
as member data (I mean, instance attributes).
--david
From se6y095@public.uni-hamburg.de Wed Sep 15 11:25:45 1999
From: se6y095@public.uni-hamburg.de (Berthold Höllmann)
Date: 15 Sep 1999 12:25:45 +0200
Subject: [Matrix-SIG] Problem with derived array dimensions in pyfort 2
Message-ID: <m2iu5c79p2.fsf@pchoel.privat.uni-hamburg.de>
Hello,
The following ".pyf" file gives me an error:
--- tst.pyf ----
module BUG
subroutine AAA(kx, ky, nxest, nyest, c)
integer kx,ky,nxest,nyest
real c ( ( ( nxest - kx - 1 ) * ( nyest - ky - 1 ) ) )
end subroutine surfit
end module BUG
--- tst.pyf ----
I get:
>pyfort -c g77 tst.pyf
Error concerning token '(' ( File 'tst.pyf', line 4 )
according to pyfort.pdf, p. 11, this syntax should be allowed.
Cheers
Berthold
--
bhoel@starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
From andy@robanal.demon.co.uk Wed Sep 15 22:49:59 1999
From: andy@robanal.demon.co.uk (Andy Robinson)
Date: Wed, 15 Sep 1999 21:49:59 GMT
Subject: [Matrix-SIG] Numeric tutorials at next conference?
Message-ID: <37e312d0.1408915@post.demon.co.uk>
I'm trying to organize a few financial talks and events at the next
Python conference. It is apparent that Python is being used very
successfully in a number of investment banks. Furthermore, I have the
impression that a few of them would send people down to learn more if
we had the right agenda. This as it is very good advertising - how
many other languages beginning with P are used to make money?
I'm hoping to persuade a few people to submit papers related to
finance, economics or accounting by the end of this month.
The icing on the cake would be for a tutorial on the Numeric
extensions (and probably for other stuff - I could pitch in and help
out on accounting). Is there anyone here who feels interested in
giving all or part of a tutorial on this? Ideally this would have a
strong applications focus, with realistic examples of things you might
acually want to do with NumPy (though not necessarily on financial
applications).
There is also a VERY high level of interest in the financial community
in an extensible event-simulation package and a time series library.
I'd like to hear about anyone working in this area with Python.
Volunteers, suggestions, ideas and flames all welcome.
Regards,
Andy Robinson
Robinson Analytics Ltd.
From godzilla@netmeg.net (Les Schaffer) Wed Sep 15 23:27:42 1999
From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer)
Date: Wed, 15 Sep 1999 18:27:42 -0400 (EDT)
Subject: [Matrix-SIG] Numeric tutorials at next conference?
In-Reply-To: <37e312d0.1408915@post.demon.co.uk>
References: <37e312d0.1408915@post.demon.co.uk>
Message-ID: <14304.7518.148972.100717@gargle.gargle.HOWL>
andy robinson said:
> There is also a VERY high level of interest in the financial
> community in an extensible event-simulation package and a time
> series library.
i'd like to hear more about the specifics of whats needed in this
area. event-simulation means what?
> Volunteers, suggestions, ideas and flames all welcome.
flames? lets see? ... down with capitalism?
les
From HYoon@exchange.ml.com Thu Sep 16 15:05:22 1999
From: HYoon@exchange.ml.com (Yoon, Hoon (CICG - NY Program Trading))
Date: Thu, 16 Sep 1999 10:05:22 -0400
Subject: [Matrix-SIG] Numeric tutorials at next conference?
Message-ID: <A1AFC4253BA3D111BFF70001FA7E9AACD62219@ewfd19.exchange.ml.com>
Les,
In my def, event simulation often means, how an event like interest rate
change, earnings announcement, yen exch rate, Indonesian crisis effects a
particular index or bench mark by how much and how long. Monte Carlo type of
simulation is not unusual. Some combined with genetic type of AI stuff. This
has close association with times seris. It take long time for one of those
effects to be work itself out. This is not my specialty, but I dabbled in
surprise unexpected earnings (SUE) model in the past, so I know little about
it. Obvisouly, SUE is a lot easier to measure than Indonesian crisis. I am
glad that's not my job - I can try the economics office upstairs for that.
I use Python for many projects that requires data structures. I am in
love with the dictionary structure. Although I am pretty much required to
use VB for implementation, I still keep many stuff in Python. Best thing for
realtime data manipulation and no data gets harder than RT data.
How about this for flame?: GODzilla sucks, King Kong rules. Even 1984, I see
cheap rubber fins.
**************************************************************
S. Hoon Yoon (Quant) Merrill Lynch
Equity Trading
yelled@yahoo.com hoon@bigfoot.com(w)
"Miracle is always only few standard deviations away, but so is
catastrophe."
* Expressed opinions are often my own, but NOT my employer's.
"I feel like a fugitive from the law of averages." Mauldin
**************************************************************
> -----Original Message-----
> From: Les Schaffer [SMTP:godzilla@netmeg.net]
> Sent: Wednesday, September 15, 1999 6:28 PM
> To: matrix-sig@python.org
> Subject: [Matrix-SIG] Numeric tutorials at next conference?
>
> andy robinson said:
>
> > There is also a VERY high level of interest in the financial
> > community in an extensible event-simulation package and a time
> > series library.
>
> i'd like to hear more about the specifics of whats needed in this
> area. event-simulation means what?
>
> > Volunteers, suggestions, ideas and flames all welcome.
>
> flames? lets see? ... down with capitalism?
>
> les
>
> _______________________________________________
> Matrix-SIG maillist - Matrix-SIG@python.org
> http://www.python.org/mailman/listinfo/matrix-sig
From hinsen@cnrs-orleans.fr Thu Sep 16 15:38:38 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Thu, 16 Sep 1999 16:38:38 +0200
Subject: [Matrix-SIG] Numerical plans (fwd)
In-Reply-To: <Pine.WNT.4.04.9909140914110.229-100000@rigoletto.ski.org>
(message from David Ascher on Tue, 14 Sep 1999 09:14:26 -0700 (Pacific
Daylight Time))
References: <Pine.WNT.4.04.9909140914110.229-100000@rigoletto.ski.org>
Message-ID: <199909161438.QAA20791@chinon.cnrs-orleans.fr>
> Well, since I think I was the main force behind that, I'll say that the
> main idea was that I wanted to be able to continue work on the
> functionality of array objects in Python, not in C. The C code is a mess,
It's a good idea to be able to implement functionality in Python.
But if the C code is a mess (and I certainly agree), it ought to be
cleaned up; there's no substitute for that, unfortunately.
> Instead, what Paul and I are planning to do is to revert to the old type,
> but to do UserArray *right*. The new features that I mention above would
> then be implemented in new Python classes, which would use array objects
> as member data (I mean, instance attributes).
But any attempt to implement significant functionality in Python
will lead to the same problem: Most users will end up using some
Python object, which C modules won't accept as an array.
My personal convention in MMTK is that all high-level "array-like"
object have an attribute "array" that stores the array object.
C modules then check for an array, and if the type check fails they
get the attribute "array" and continue with that. Maybe something
similar would be a good convention in general.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From da@ski.org Thu Sep 16 17:17:52 1999
From: da@ski.org (David Ascher)
Date: Thu, 16 Sep 1999 09:17:52 -0700 (Pacific Daylight Time)
Subject: [Matrix-SIG] Numerical plans (fwd)
In-Reply-To: <199909161438.QAA20791@chinon.cnrs-orleans.fr>
Message-ID: <Pine.WNT.4.04.9909160917040.200-100000@rigoletto.ski.org>
> > Instead, what Paul and I are planning to do is to revert to the old type,
> > but to do UserArray *right*. The new features that I mention above would
> > then be implemented in new Python classes, which would use array objects
> > as member data (I mean, instance attributes).
>
> But any attempt to implement significant functionality in Python
> will lead to the same problem: Most users will end up using some
> Python object, which C modules won't accept as an array.
>
> My personal convention in MMTK is that all high-level "array-like"
> object have an attribute "array" that stores the array object. C
> modules then check for an array, and if the type check fails they get
> the attribute "array" and continue with that. Maybe something similar
> would be a good convention in general.
That's exactly what I had in mind. We'd just be using UserArray as the
prototype for others to follow.
--david
From amullhau@zen-pharaohs.com Thu Sep 16 17:26:12 1999
From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt)
Date: Thu, 16 Sep 1999 12:26:12 -0400
Subject: [Matrix-SIG] Numeric tutorials at next conference?
References: <37e312d0.1408915@post.demon.co.uk> <14304.7518.148972.100717@gargle.gargle.HOWL>
Message-ID: <016301bf0066$b1fbda00$a602550a@amullhau>
> andy robinson said:
>
> > There is also a VERY high level of interest in the financial
> > community in an extensible event-simulation package and a time
> > series library.
>
> i'd like to hear more about the specifics of whats needed in this
> area. event-simulation means what?
Event simulation is basically a term for simulation of processes in which
there are contingent events. For example, if you wanted to simulate a
grocery store, you might have events like "customer enters aisle 6" and "box
of Fruit Loops taken from shelf and put in customer's cart", etc.
Event simulation support usually means a language in which you can write the
events and dependencies in a reasonably simple way and then a simulation can
be run. Simulation languages go back a long way (Simula is an example) and
people who use them love them.
It makes sense if you don't care about high performance, but not really
otherwise. Really large scale simulations usually can be written, and have
to be written, outside the event simulation paradigm.
Later,
Andrew Mullhaupt
From guido@CNRI.Reston.VA.US Thu Sep 16 19:53:52 1999
From: guido@CNRI.Reston.VA.US (Guido van Rossum)
Date: Thu, 16 Sep 1999 14:53:52 -0400
Subject: [Matrix-SIG] Reminder: Python Conference papers due soon!
Message-ID: <199909161853.OAA01738@eric.cnri.reston.va.us>
The deadline for paper submissions for the next Python conference is
nearing! Make sure that if you are considering submitting a paper,
the program chair sees your paper by SEPTEMBER 30. Papers can be on
any Python, JPython or Zope related subject. Papers will be reviewed
and the best papers will be selected for presentation. You will hear
about selection by October 22.
For more information, see the official call for papers:
http://www.python.org/workshops/2000-01/cfp.html
** We are planning a significant Zope presence at this conference! **
The conference will be held from January 24-27 in Alexandria, VA (just
across the Potomac from DC). Make Python the first conference in the
new millennium you visit! For more info, see the conference home
page: http://www.python.org/workshops/2000-01/
Send all email questions about papers to: ipc8papers@cs.uchicago.edu
--Guido van Rossum (home page: http://www.python.org/~guido/)
(And sorry for the broad spamming.)
From se6y095@public.uni-hamburg.de Sun Sep 19 19:38:25 1999
From: se6y095@public.uni-hamburg.de (Berthold Höllmann)
Date: 19 Sep 1999 20:38:25 +0200
Subject: [Matrix-SIG] Pyfort: Intent inout must be an array.? / Please make meaningfull exit values.
Message-ID: <m29062d9we.fsf@pchoel.privat.uni-hamburg.de>
Hello,
I try to wrap a FORTRAN function which uses one integer parameter as
input as well as as output value, so I tried:
integer, intent(inout):: n
but pyfort quits with the message:
('Intent inout must be an array.',)
is there a technical reason for this, or is this only not implemented.
Another point I have is, that if an error occurs in pyfort it uses
raise SystemExit
in some places, I would prefer to make it
raise SystemExit, 1
becaus this would allow make to stop at this point and not trying to
proceed.
Cheers
Berthold
--
bhoel@starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
From dubois1@llnl.gov Sun Sep 19 20:29:13 1999
From: dubois1@llnl.gov (Paul F. Dubois)
Date: Sun, 19 Sep 1999 12:29:13 -0700
Subject: [Matrix-SIG] Pyfort: Intent inout must be an array.? / Please make meaningfull exit values.
References: <m29062d9we.fsf@pchoel.privat.uni-hamburg.de>
Message-ID: <000f01bf02d5$41b16e60$3c810b18@plstn1.sfba.home.com>
----- Original Message -----
From: Berthold Höllmann <se6y095@public.uni-hamburg.de>
To: <matrix-sig@python.org>
Cc: <dubois1@llnl.gov>
>
> I try to wrap a FORTRAN function which uses one integer parameter as
> input as well as as output value, so I tried:
>
> integer, intent(inout):: n
>
> but pyfort quits with the message:
>
> ('Intent inout must be an array.',)
>
> is there a technical reason for this, or is this only not implemented.
>
The user must pass an array since Python integers are immutable. It would of
course be possible to write the wrapper and then object at runtime if the
argument passed wan't an array; but then the documentation would not match
what the user had to do.
I chose to do this to make the point crystal clear. The fix is to change it
to n(1) in your Pyfort input. You need not touch the Fortran, of course. I
could be talked out of this decision by someone with a forceful personality;
it was an arbitrary decision.
> Another point I have is, that if an error occurs in pyfort it uses
>
> raise SystemExit
>
> in some places, I would prefer to make it
>
> raise SystemExit, 1
>
> becaus this would allow make to stop at this point and not trying to
> proceed.
>
Will fix.
From minxu@sci.ccny.cuny.edu Sun Sep 19 20:34:32 1999
From: minxu@sci.ccny.cuny.edu (minxu@sci.ccny.cuny.edu)
Date: Sun, 19 Sep 1999 15:34:32 -0400
Subject: [Matrix-SIG] HELP! How to access a large array
Message-ID: <199909191934.PAA21699@star.future.com>
Hi,
I am currently implementing a image reconstruction code with Python. It goes
smoothly but I am stuck by one problem.
One part of my code calculates some large matrices around 24M+ and stored in
netcdf file. The second part wants to read in some data from the created
netcdf file and do processing accordingly.
But is turns out that it takes minutes for the second part to read the data
from netcdf file. It is very sad since I want to increase the size of data
further.
Is there any way that I can read some subsets of one array stored in netcdf
file without loading the whole array? Does f.variables['var'][x, y, ...] avoid
loading the whole 'var' array?
If netcdf can not do that, any other storage can do that?
Thanks.
--
Min Xu
City College of NY, CUNY
Email: mxu1@email.gc.cuny.edu
minxu@sci.ccny.cuny.edu
Tel: (O) (212) 650-6865
(O) (212) 650-5046
(H) (212) 690-2119
From jhauser@ifm.uni-kiel.de Mon Sep 20 13:15:38 1999
From: jhauser@ifm.uni-kiel.de (Janko Hauser)
Date: Mon, 20 Sep 1999 14:15:38 +0200 (CEST)
Subject: [Matrix-SIG] HELP! How to access a large array
In-Reply-To: <199909191934.PAA21699@star.future.com>
References: <199909191934.PAA21699@star.future.com>
Message-ID: <14310.9578.229015.356053@ifm.uni-kiel.de>
minxu@sci.ccny.cuny.edu writes:
> But is turns out that it takes minutes for the second part to read the data
> from netcdf file. It is very sad since I want to increase the size of data
> further.
>
Along which dimension are you doing the subsampling. To read
subsamples along the first (unlimited) dimension is very timeconsuming
but not memory consuming. Do you see, that all the data is read in at
the memory footprint? Or do you asume this because of the long time it
took. It should not read in all the data. Which version of the
netcdf-library are you using? The 2.? is slower then the newer ones.
> Is there any way that I can read some subsets of one array stored in netcdf
> file without loading the whole array? Does f.variables['var'][x, y, ...] avoid
> loading the whole 'var' array?
>
This should already be possible? I deal with NetCDF-files of ca. 1Gb
size, so this is working.
__Janko
From jhauser@ifm.uni-kiel.de Mon Sep 20 13:21:32 1999
From: jhauser@ifm.uni-kiel.de (Janko Hauser)
Date: Mon, 20 Sep 1999 14:21:32 +0200 (CEST)
Subject: [Matrix-SIG] PyFort: Other function names.
Message-ID: <14310.9932.8635.147745@ifm.uni-kiel.de>
Used for the first time the PyFort module. This is very slick and
elegant and working out of the box. I want to wrap a function, where
an integer parameter defines if the interpolation is done linear or
cubic. This affects also the length of the working array, which I give
as a temporary.
Is it possible to give a new name for the wrapped function and thus to
wrap the same function with different interfaces? This would also
allow to give sometimes better names to cryptic Fortran names. Alos
the use of a temporary could then be more flexibel (have a fast one,
where always the same temporary is used, but which needs to be
provided by the interpreter)
Thanks for this tool,
__Janko
From minxu@scisun.sci.ccny.cuny.edu Mon Sep 20 16:57:41 1999
From: minxu@scisun.sci.ccny.cuny.edu (Min Xu)
Date: Mon, 20 Sep 1999 11:57:41 -0400
Subject: [Matrix-SIG] HELP! How to access a large array
In-Reply-To: Message from Janko Hauser <jhauser@ifm.uni-kiel.de>
of "Mon, 20 Sep 1999 14:15:38 +0200." <14310.9578.229015.356053@ifm.uni-kiel.de>
Message-ID: <199909201557.LAA01533@star.future.com>
> minxu@sci.ccny.cuny.edu writes:
> > But is turns out that it takes minutes for the second part to read the data
> > from netcdf file. It is very sad since I want to increase the size of data
> > further.
> >
>
> Along which dimension are you doing the subsampling. To read
> subsamples along the first (unlimited) dimension is very timeconsuming
> but not memory consuming. Do you see, that all the data is read in at
> the memory footprint? Or do you asume this because of the long time it
> took. It should not read in all the data. Which version of the
> netcdf-library are you using? The 2.? is slower then the newer ones.
>
I try to load the *whole* netcdf array in the second part as my first attempt.
The netcdf array is:
['w', 'mf_i', 'mf_r'], double and their sizes are:
>>> f.mf_r.shape
(13, 18, 18, 38, 38)
>>> f.mf_i.shape
(13, 18, 18, 38, 38)
>>> f.wf.shape
(10, 13, 10, 10, 20, 20)
The codelet is here:
def getdata(self):
f = netcdf.NetCDFFile("gkxy.nc", "r")
for m in self.c.MEASTYPE:
print 'Allocating for', m
start = time()
self.m[m] = zeros(f.variables[self.mmap[(m,
self.c.SAMPTYPE[0])]+'_r'].shape, 'D')
print 'time elapsed:', time() - start
for s in self.c.SAMPTYPE:
print 'Reading meas for', m, s
start = time()
self.m[m].real = self.m[m].real + \
f.variables[self.mmap[(m, s)]+'_r'][:]
self.m[m].imaginary = self.m[m].imaginary + \
f.variables[self.mmap[(m, s)]+'_i'][:]
print 'time elasped:', time() - start
print 'Reading weight for', m, s
start = time()
self.w[(m, s)] = f.variables[self.wmap[(m, s)]][:]
print 'time elapsed:', time() - start
f.close()
What I got is:
>>> i.getdata()
Allocating for 1
time elapsed: 1.53462100029
Reading meas for 1 1
time elasped: 185.918686032
Reading weight for 1 1
time elapsed: 3.38862001896
>>>
Although the size of meas and weight is comparable, the time ratio is about
*60*. The memory of my SGI200 machine is 256M but CPU is idle (<1% busy) and
waiting for SWAP (>97% ratio) most of time when reading meas. Some 20M free
memory is reported at the same time.
netcdf library is 3.4 and the python module is Hinsen's.
Can you tell me some reasons for that?
> > Is there any way that I can read some subsets of one array stored in netcdf
> > file without loading the whole array? Does f.variables['var'][x, y, ...] avoid
> > loading the whole 'var' array?
> >
> This should already be possible? I deal with NetCDF-files of ca. 1Gb
> size, so this is working.
>
I agree. Since the second part has to read hundred of times of different
subsets of the netcdf array and almost covers the whole array, I wonder it may
be better to read the whole array at once.
Do you have a good method to handle complex data in netcdf?
--
Min Xu
City College of NY, CUNY
Email: mxu1@email.gc.cuny.edu
minxu@sci.ccny.cuny.edu
Tel: (O) (212) 650-6865
(O) (212) 650-5046
(H) (212) 690-2119
From hinsen@cnrs-orleans.fr Mon Sep 20 20:17:44 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Mon, 20 Sep 1999 21:17:44 +0200
Subject: [Matrix-SIG] HELP! How to access a large array
In-Reply-To: <199909191934.PAA21699@star.future.com> (minxu@sci.ccny.cuny.edu)
References: <199909191934.PAA21699@star.future.com>
Message-ID: <199909201917.VAA13937@chinon.cnrs-orleans.fr>
> Is there any way that I can read some subsets of one array stored in netcdf
> file without loading the whole array? Does f.variables['var'][x, y, ...] avoid
> loading the whole 'var' array?
Yes. Since many netCDF files store arrays that are larger than
physical memory, this is in fact often the only way to deal with data
in netCDF files. Any access to a subarray will read precisely that
subarray. However, the time required for such an operation also
depends on the layout of the data in the file.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From perry@stsci.edu Tue Sep 21 20:07:31 1999
From: perry@stsci.edu (Perry Greenfield)
Date: Tue, 21 Sep 1999 15:07:31 -0400 (EDT)
Subject: [Matrix-SIG] Re: Numeric plans
Message-ID: <199909211907.PAA06724@eclipse.stsci.edu>
We have a few comments on some of the points made in the past week or so
regarding future work on Numeric.
1) "[Numeric] C code is a mess." (David Asher)
Could David, Paul or someone elaborate a bit on what this means? Is
this meant to say that it is just a mess to modify, or (worse) that it
is unreliable as is? If the latter, it would seem to argue for a
concerted effort to clean up soon; if the former, it would still argue
for clean up, but perhaps on a more relaxed time scale. What scale of
effort would be involved?
2) Paul Dubois mentioned the problems he was having settling on a
consensus for how the "inverse of take" (the old scatter/gather issue)
should be implemented when the array being indexed has rank > 1.
> (Periodically the issue is raised of
>
> x[indices] = values
> x[indices] as an r-value
>
> being desireable, as it was recently. As I pointed out before, everyone
> agrees that the above line is desireable, but when x has rank > 1, there
> are strongly differing views of what this ought to mean. Multiple
> intelligent, experienced developers have told me that (a) I am an idiot,
> because (b) there is only one obvious and highly desireable solution,
> namely theirs Unfortunately the latter does not turn out to be the same
> in all cases.
>
> The issue is complicated by the fact that the coding for setindex is
> already totally incomprehensible and so changing it requires a great
> deal of care and testing.
>
> It is time for proposals. Concrete, detailed. Discussion to follow. If
> we can come to a consensus it would be nice.
> Do not forget things like NewAxis, Ellipsis, and the colon. How do these
> things mix and match?
We think (as we've said before) that a less general mechanism would be
better than none if consensus can't be achieved. In other words, do
what Andrew Mullhaupt suggested and either require x to be rank 1 or
coerce x to rank 1, producing a rank 1 result. The advantage of the
former is that if we eventually can agree on what should be done with
rank>1, then anything we decide will be backward compatible. Assuming
that everyone agrees on what this means for a rank-1 array x, that
behavior could be added immediately.
Perhaps Paul (or someone) could summarize the competing views on this
issue to spur some discussion on the more general approach. There is a
fairly large number of past messages about this spread over years. A
couple page summary of the competing proposals with the pros and cons
of each (that we are aware of) would help focus the discussion. Maybe
there will be consensus (or a one that a clear majority favors). Paul,
is it possible to summarize this?
3) Are there aspects of this work (UserArray, general code clean-ups,
or inverse of take) that others can help contribute to? If not...
4) A means of inheriting Numeric arrays is important to us also and we
are very interested in how that issue is going to be solved. Is there
any way for those of us that are interested to keep abreast of the ongoing
design and implementation discussions?
Perry Greenfield
Rick White
Paul Barrett
From dubois1@llnl.gov Wed Sep 22 00:07:38 1999
From: dubois1@llnl.gov (Paul F. Dubois)
Date: Tue, 21 Sep 1999 16:07:38 -0700
Subject: [Matrix-SIG] Re: Numeric plans
References: <199909211907.PAA06724@eclipse.stsci.edu>
Message-ID: <001701bf0486$34862ac0$3c810b18@plstn1.sfba.home.com>
This is a multi-part message in MIME format.
------=_NextPart_000_000B_01BF044B.6D03FF20
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Perry et. al asked me for comments on the following:
> 1) "[Numeric] C code is a mess." (David Asher)
I invite you to peruse the source. Imagine you want to make the change
requested in #2.
I believe you will then hold this truth to be self-evident.
>
> 2) Paul Dubois mentioned the problems he was having settling on a
> consensus for how the "inverse of take" (the old scatter/gather issue)
> should be implemented when the array being indexed has rank > 1.
...
> Paul,
> is it possible to summarize this?
Paul has a day job. That is why he asked for proposals. If your proposal is
to implement x[indices] as both an r- and l-value for x one-dimensional
only, then by golly I do know what to do. But I don't necessarily know how
to do it, see #1. Egads, a circular reference, I will never be garbage
collected.
>
> 3) Are there aspects of this work (UserArray, general code clean-ups,
> or inverse of take) that others can help contribute to? If not...
People have been contributing when the spirit moved them by sending in
patches. David or I review the patches and as long as it doesn't offend our
sensibilities we put them in. We'd like to get #12 fixed after our screwup
and then would welcome things such as #2 being done for us. We both have a
lot on our plates so we are in the annoying position of locking out the
volunteer help for the moment because we are too busy.
>
> 4) A means of inheriting Numeric arrays is important to us also and we
> are very interested in how that issue is going to be solved. Is there
> any way for those of us that are interested to keep abreast of the ongoing
> design and implementation discussions?
David and I finally got a chance to talk together in person for a couple of
hours and we had some ideas about how to proceed. That is the sum total of
the "design". The general idea is to make a decent UserArray; this will
require adding some things to the core. We got a lot of experience during
the recent wild goose chase. I've been working on a facility for what I call
Masked Arrays; that is, arrays that have an optional mask that indicates
missing values. This *is* part of my day job so it is getting into pretty
good shape now. I was going to back-fit UserArray from it. I attach the
current version for your comments. This incarnation tries to let Numeric
arrays and these MA arrays coexist.
>
> Perry Greenfield
> Rick White
> Paul Barrett
>
Supposing there were a full-featured UserArray module, similar to MA but
without the mask, would it be helpful to you? Or perhaps, would it be
helpful as a reuse-by-editor form of inheritance? I guess what I think I
found out is that it is almost not really possible to inherit from Numeric
without fairly massive redefinitions. There are some tricks one might be
able to play to make a few extensions possible, such as using self.__class__
as a constructor where possible, but maybe even that isn't a good idea since
it assumes you can make a valid Foo using just data as an argument, clearly
not true in general.
For the longer term, it would be nice to get a do-over. However, there are
many different communities here with different needs and few social skills
with which to resolve the differences. (:->. This fact is easiest to see if
you think about x[i, :]. The speed burners want this to be a reference, the
usabilities want this to be a copy.
Best regards,
Paul
------=_NextPart_000_000B_01BF044B.6D03FF20
Content-Type: text/plain;
name="MA.py"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="MA.py"
"""MA facility Copyright 1999 Regents of the University of California.
Released for unlimited redistribution; see file Legal.htm in Numerical =
distribution."""
__version__ =3D 2.0
#This version gives up on the Extension Class stuff
import Numeric
from Precision import *
import string, types
class MAError (Exception):
def __init__ (self, args=3DNone):
self.args =3D args
def __str__(self):
return str(self.args)
__MaxElements =3D 300 #Maximum size for printing
def setMaxElements (m):
"Set the maximum # of elements for printing arrays. "
global __MaxElements
n =3D int(m)
if n <=3D 0: raise ValueError, "setMaxElements requires positive =
argument."
__MaxElements =3D n
def getMaxElements ():
"Get the maximum # of elements for printing arrays. "
global __MaxElements
return __MaxElements
def getmask (a):
"Mask of values in a, if any; None otherwise."
if isMA(a):=20
return a.mask()
else:=20
return None
def getmaskarray (a):
"Mask of values in a, ones if none."
m =3D getmask(a)
if m is None:
return Numeric.ones(shape(a))
else:
return m
def mask_or (m1, m2):
"Logical or of the masks m1 and m2, treating None as false."
if m1 is None: return m2
if m2 is None: return m1
return Numeric.logical_or(m1, m2)
def filled (a, value =3D None):
"""
a as a Numeric array with any masked areas replaced by value"
No data copied if no masked area and data is already an array.
"""
if isMA(a):
return a.filled(value)
else:
return Numeric.array(a, copy=3D0)
class masked_unary_operation:
def __init__ (self, aufunc, fill=3D0):
self.f =3D aufunc
self.fill =3D fill
def __call__ (self, a):
m =3D getmask(a)
d1 =3D filled(a)
if m is None:
return self.f(d1)
else:
return MA(self.f(d1), m)
class masked_binary_operation:
def __init__ (self, aufunc, fill=3D0):
self.f =3D aufunc
self.fill =3D fill
def __call__ (self, a, b):
d1 =3D filled(a, self.fill)
d2 =3D filled(b, self.fill)
m =3D mask_or(getmask(a), getmask(b))
if m is None:
return self.f(d1, d2)
else:
return MA(self.f(d1,d2), m)
def reduce (self, target, axis=3D0):
m =3D getmask(target)
if m is None:
return self.f.reduce (target, axis)
else:
t =3D target.filled(self.fill)
tr =3D self.f.reduce(t, axis)
m =3D Numeric.logical_and.reduce(m, axis)
return MA(tr, m)
class comparison:
"A comparison function that returns a plain mask after doing the =
comparison"
def __init__ (self, f, fill1, fill2):
self.f =3D f
self.fill1 =3D fill1
self.fill2 =3D fill2
def __call__ (self, a, b):
return self.f(filled(a, self.fill1), filled(b, self.fill2))
# class MA =20
# the use of filled in the constructor
# makes sure we don't get MA's in data or mask=20
# This prevents recursion
# and we can assume these are array objects (or None for mask).
# Note current implementation assumes mask is 1's and 0's
# so that we can use Numeric.choose in filled.
class MA:
"Arrays with possibly masked values."
default_real_fill_value =3D 0.0
default_complex_fill_value =3D 0.0 + 0.0j
default_character_fill_value =3D ' '
default_integer_fill_value =3D 0
default_unsigned_integer_fill_value =3D 0
default_object_fill_value =3D None
def __init__(self, data, mask=3DNone, **keys):
"""MA(data, mask=3DNone)"""
self.__data =3D filled(data)
if mask is None:
self.__mask =3D None
else:
self.__mask =3D filled(mask, 1)
def __getattr__ (self, name):
if name =3D=3D 'shape':
return self.get_shape()
else:
return self.__dict__[name]
def __setattr__ (self, name, value):
if name =3D=3D 'shape':
self.set_shape(value)
else:
self.__dict__[name] =3D value
def __str__(self):
return str(self.filled())
def __repr__(self):
if self.__mask is None:
return "MA(" + repr(filled(self)) + ")"
else:
return "MA(" + repr(filled(self)) + ", \n " + \
repr(self.__mask) + ")"
def __float__(self):
return MA (float(self.filled(0.0)), self.__mask)
def __int__(self):
return MA(int(self.filled(0)), self.__mask)
# Note copy semantics here differ from Numeric =20
def __getitem__(self, i):=20
m =3D self.__mask
if m is None:
return Numeric.array(self.__data[i])
else:
return MA(Numeric.array(self.__data[i]), =
Numeric.array(m[i]))
def __getslice__(self, i, j):=20
m =3D self.__mask
if m is None:
return Numeric.array(self.__data[i:j])
else:
return MA(Numeric.array(self.__data[i:j]), =
Numeric.array(m[i:j]))
# --------
def __setitem__(self, index, value):
self.__data[index] =3D filled(value)
self.__mask =3D mask_or(self.__mask, getmask(value))
=20
def __setslice__(self, i, j, value):
self.__data[i:j] =3D filled(value)
m =3D getmask(value)
if m is not None:
sm =3D getmaskarray(self)
sm[i:j] =3D mask_or(sm[i:j], m)
self.__mask =3D sm
def __len__ (self):
"Return total number of elements in array."
return self.size()
def __cmp__ (self, other):
raise MAError, "Cannot use comparison operators on MA =
instances."
=20
def __abs__(self):=20
return masked_unary_operation(Numeric.absolute)(self)
def __neg__(self):=20
return masked_unary_operation(Numeric.negative)(self)
def __add__(self, other):=20
return masked_binary_operation(Numeric.add)(self, other)
=20
__radd__ =3D __add__
def __sub__(self, other):=20
return masked_binary_operation(Numeric.subtract,0)(self, other)
def __rsub__(self, other):=20
return masked_binary_operation(Numeric.subtract,0)(other, self)
def __mul__(self, other):=20
return masked_binary_operation(Numeric.multiply,0)(self, other)
=20
__rmul__ =3D __mul__
def __div__(self, other):=20
return divide(self, other)
def __rdiv__(self, other):=20
return divide(other, self)
def __pow__(self,other, third=3DNone):=20
if third is not None:=20
raise MAException, "3 arg power unspoorted"
return masked_binary_operation(Numeric.power, 1)(self, other)
def __sqrt__(self):=20
return masked_unary_operation(Numeric.sqrt,0)(self)
def default_fill_value (self):
"Function to calculate default fill value for an array."
x =3D self.typecode()
if x in typecodes['Float']:
return MA.default_real_fill_value
if x in typecodes['Integer']:
return MA.default_integer_fill_value
if x in typecodes['Complex']:
return MA.default_complex_fill_value
if x in typecodes['Character']:
return MA.default_character_fill_value
if x in typecodes['UnsignedInteger']:
return Numeric.absolute(MA.default_integer_fill_value)
else:
return MA.default_object_fill_value
def set_fill_value (self, v):
"Set the fill value to v."
self.fill_value =3D v
def filled (self, value=3DNone):
""" Self with masked data replaced by value.
If value is None, self.fill_value used.
self.default_fill_value () used if value is None.
Not a copy if no masked data.
Result is romised to be a Numeric.array.
"""
if self.__mask is None:
return self.__data
else:
if value is None:
if hasattr(self, 'fill_value'):
v =3D self.fill_value
else:
v =3D self.default_fill_value ()
self.fill_value =3D v
else:
v =3D value
return Numeric.choose(self.__mask, (self.__data, v))
def get_shape(self):
"Return the tuple giving the current shape."
return self.__data.shape
def ids (self):
"""Return the ids of the data and mask areas"""
return (id(self.__data), id(self.__mask))
def mask(self):
"Return the data mask, or None."
return self.__mask
=20
def set_shape (self, *sizes):
"""shape(n, m, ...) sets the shape."""
self.__data.shape =3D sizes
if self.__mask is not None:
self.__mask.shape =3D sizes
=20
def size (self, axis =3D None):
"Number of elements in array, or in a particular axis."
s =3D self.shape
if axis is None:
return reduce(Numeric.multiply, s, 1)
else:
return s[axis]
=20
def typecode(self):
return self.__data.typecode()
# these are disgusting
def is_contiguous (self):
return self.data.is_contiguous()
def byte_swapped(self):
return self.data.byte_swapped()
=20
def isMA (x):
"Is x an instance of MA?"
return isinstance(x, MA)
def masked_array (data, masked_value, typecode=3DNone, copy=3D1, =
atol=3D1.e-8, rtol=3D1.e-8, notest=3D0):
"""Create a masked array for numerical data; no mask unless =
necessary
or if notest !=3D 0
"""
abs =3D Numeric.absolute
d =3D Numeric.array(data, typecode=3Dtypecode, copy=3Dcopy)
dm =3D Numeric.less(abs(d-masked_value), atol+rtol*abs(d))
if notest or Numeric.sometrue(Numeric.ravel(dm)):
result =3D MA(d, dm)
else:
result =3D MA(d)
result.set_fill_value(masked_value)
return result
=20
def masked_object (data, masked_value, copy=3D1):
"Create a masked array of places where exactly equal to masked =
value"
dm =3D Numeric.equal(data, masked_value, copy=3Dcopy)
return MA(data, dm)
=20
def rank (a):
"Get the rank of a"
return len(shape(a))
=20
def shape (a):
"Get the shape of a"
if isMA (a):
return a.get_shape()
else:
return Numeric.shape(a)
def size (a, axis=3DNone):
"Get the number of elements in a, or along a certain axis."
if isMA (a):
return a.size(axis)
else:
s =3D Numeric.shape(a)
if axis is None:
return reduce(Numeric.multiply, s, 1)
else:
return s[axis]
def count (a, axis =3D None):
"Count of the non-masked elements." =20
m =3D getmask(a)
if m is None:
return size(a, axis)
else:
n1 =3D size(a, axis)
n2 =3D sum (m, axis)
return n1 - n2
def sum (a, axis =3D None):
"Sum of all elements, or along a certain axis"
if axis is None:
return Numeric.add.reduce(Numeric.ravel(filled(a, 0)))
else:
return Numeric.add.reduce(filled(a, 0), axis)
=20
def product (a, axis =3D None):
"Product of all elements, or along a certain axis."
if axis is None:
return Numeric.add.reduce(Numeric.ravel(filled(a, 0)))
else:
return Numeric.add.reduce(filled(a, 0), axis)
def average(a, axis=3DNone):
"Average of the nonmasked elements of a"
c =3D 1.0 * count(a, axis)
a =3D sum(a, axis)=20
return divide(a, c)
=20
NewAxis =3D Numeric.NewAxis
arange =3D Numeric.arange
ones =3D Numeric.ones
zeros =3D Numeric.zeros
array =3D Numeric.array
#minimum, maximum?
#searchsorted?
sqrt =3D masked_unary_operation(Numeric.sqrt, 0.0)
sin =3D masked_unary_operation(Numeric.sin, 0.0)
cos =3D masked_unary_operation(Numeric.cos, 0.0)
absolute =3D masked_unary_operation(Numeric.absolute, 0)
negative =3D masked_unary_operation(Numeric.negative, 0)
absolute =3D masked_unary_operation(Numeric.absolute, 0)
nonzero =3D masked_unary_operation(Numeric.nonzero, 0)
add =3D masked_binary_operation(Numeric.add, 0)
subtract =3D masked_binary_operation(Numeric.subtract, 0)
multiply =3D masked_binary_operation(Numeric.multiply, 1)
power =3D masked_binary_operation(Numeric.power, 1)
sometrue =3D masked_unary_operation(Numeric.sometrue, 0)
alltrue =3D masked_unary_operation(Numeric.alltrue, 1)
# all these comparisons return false where masks are true =20
equal =3D comparison(Numeric.equal, 0, 1)
not_equal =3D comparison(Numeric.not_equal, 0, 0)
less_equal =3D comparison(Numeric.less_equal, 1, 0)
greater_equal =3D comparison(Numeric.greater_equal, 0, 1)
less =3D comparison(Numeric.less, 1, 0)
greater =3D comparison(Numeric.greater, 0, 1)
def choose (condition, t):
"Shaped like condition, values t[0] where condition true, t[1] =
otherwise"
d =3D Numeric.choose(filled(condition,0), map(filled, t))
m =3D Numeric.choose(filled(condition,0), map(getmaskarray, t))
return MA(d, m)
def where (condition, x, y):
return choose(Numeric.not_equal(condition, 0), (y, x))
def reshape (a, newshape):
"Copy of a with a new shape."
m =3D getmask(a)
d =3D Numeric.reshape(filled(a), newshape)
if m is None:
return d
else:
return MA(d, Numeric.reshape(m, newshape))
def ravel (a):
"a as one-dimensional, may share data and mask"
m =3D getmask(a)
d =3D Numeric.ravel(filled(a)) =20
if m is None:
return d
else:
return MA(d, Numeric.ravel(m))
def concatenate (somearrays, axis=3D0):
"concatenate the arrays along the given axis"
d =3D Numeric.concatenate(map(filled, somearrays), axis)
dm =3D Numeric.concatenate(map(getmaskarray, somearrays), axis)
return MA(d, dm)
def take (a, indices, axis=3D0):
m =3D getmask(a)
d =3D filled(a)
if m is None:
return Numeric.take(d, indices, axis)
else:
return MA(Numeric.take(d, indices, axis),=20
Numeric.take(m, indices, axis))
=20
def divide (x, y):
"a/b, masked where b was masked or zero"
a =3D filled(x, 0)
b =3D filled(y, 1)
bad_elements =3D equal(b, 0)
c_mask =3D mask_or(mask_or(getmask(a), getmask(b)), bad_elements)
return MA(Numeric.divide(a, b), c_mask)
=20
# This section is stolen from a post about how to limit array printing.
def limitedArrayRepr(a, max_line_width =3D None, precision =3D None, =
suppress_small =3D None):
global __MaxElements
s =3D a.shape
elems =3D Numeric.multiply.reduce(s)
if elems > __MaxElements:
if len(s) > 1:
return 'array (%s) , type =3D %s, has %d elements' % \
(string.join(map(str, s), ","), a.typecode(), elems)
else:
return Numeric.array2string (a[:__MaxElements], =
max_line_width, precision,
suppress_small,',',0) + \
('\n + %d more elements' % (elems - __MaxElements))
else:
return Numeric.array2string (a, max_line_width, precision,=20
suppress_small,',',0)
# replace the default with a smarter function for huge arrays
original_array_repr =3D Numeric.array_repr
original_array_str =3D Numeric.array_str
Numeric.array_repr =3D limitedArrayRepr
Numeric.array_str =3D limitedArrayRepr
# fixup multiarray, as well.
import multiarray
multiarray.set_string_function(limitedArrayRepr, 0)
multiarray.set_string_function(limitedArrayRepr, 1)
if __name__ =3D=3D '__main__':
import MAtest
------=_NextPart_000_000B_01BF044B.6D03FF20
Content-Type: text/plain;
name="MAtest.py"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="MAtest.py"
from MA import *
def put(*stuff):
for x in stuff:
print x,
print
def test (va, mv):
ua =3D masked_array(va, mv)
put("shape(ua), ua.shape, ua.get_shape()", shape(ua), ua.shape, =
ua.get_shape())
put("rank(ua), size(ua), size(ua,0), size(ua,1)\n",=20
rank(ua), size(ua), size(ua,0), size(ua,1))
put("mask(ua)\n", getmask(ua))
put("repr(ua)\n", repr(ua))
put("va\n", va)
put("ua\n", ua)
put ("ua.filled(-1)\n", ua.filled(-1))
put("isMA(ua)", isMA(ua))
put("isMA(va)", isMA(va))
put("ua * va", type(ua*va), '\n', ua*va)
put("va * ua", type(va*ua), '\n', va*ua)
put("ua[:,2]\n", ua[:,2])
u1 =3D ravel(ua)
put("u1", u1.filled(), u1.mask())
=20
put("u1[7]\n", u1[7])
put("u1[4:12]\n", u1[4:12])
put("cos(ua)\n", cos(ua))
put("abssolute(-ua)\n", absolute(-ua))
put("ua+ua\n", ua+ua)
put("ua**2\n", ua**2)
put("sum(ua)\n", sum(ua))
put("sum(ua, 1)\n", sum(ua, 1))
put("equal(ua, va)\n", equal(ua, va))
put("average(ua)\n", average(ua))
put("average(ua, 1)\n", average(ua, 1))
put("take(ua, (1,2))\n", take(ua, (1,2)))
put("take(ua, (1,2), 1)\n", take(ua, (1,2), 1))
va=3Dreshape(arange(12), (4,3))
mv =3D 99
va[0,1]=3Dmv
va[2,2] =3D mv
test (va, mv)
va =3D reshape(arange(12)*1.0, (4,3))
mv =3D 1.e20
va[0,1] =3D mv
va[2,2] =3D mv
test (va, mv)
# now try pathology
va=3Dreshape(arange(12)*1.0, (4,3))
mv =3D 1.e20
va[0,1] =3D mv
va[2,2] =3D mv
va[:, 2] =3D mv
test (va, mv)
a =3D arange(4, typecode=3DFloat)
b =3D arange(4, typecode=3DFloat) -2.0
am =3D MA(a, [1,0,0,0])
bm =3D MA(b, [0,0,0,1])
put("am\n", am)
put("bm\n", bm)
put("a\n", a)
put("b\n", b)
put("divide(a, b)\n", divide(a, b))
put("am / b\n", am / b)
put("a / bm\n", a / bm)
put("am / bm\n", am / bm)
ua =3D masked_array(va, mv, copy=3D0)
ub =3D MA(va, ua.mask()) #shares a mask and data
ua[1,0] =3D 22.0
uc =3D MA(ua, ua.mask()) #shares only the mask
ua.mask()[0,0] =3D 1
put("ids\n", map(lambda x: x.ids(), (ua,ub,uc)))
put("3 tests of sharing, should be all zeros\n")
put("ub[1,0] - ua[1,0]\n", ub[1,0] - ua[1, 0])
put("ua.filled(10.) - ub.filled(10.)\n", ua.filled(10.) - =
ub.filled(10.))
put("ua - uc\n", ua - uc)
wo =3D where([1,1,0,0], am, am*10.)
wo.missing_value =3D 1.e20
put("wo\n", wo)
ns =3D arange(12, typecode=3DFloat)
ms =3D masked_array(ns, 5.0)
qs =3D masked_array(ns, 100.0)
ns =3D reshape(ns, (4,3))
ms.set_shape(4,3)
qs.set_shape(4,3)
put("Should be (4,3)\n", shape(ns))
put("Should be (4,3)\n", shape(ms))
put("Should be (4,3)\n", shape(qs))
------=_NextPart_000_000B_01BF044B.6D03FF20--
From amullhau@zen-pharaohs.com Wed Sep 22 02:45:01 1999
From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt)
Date: Tue, 21 Sep 1999 21:45:01 -0400
Subject: [Matrix-SIG] Re: Numeric plans
References: <199909211907.PAA06724@eclipse.stsci.edu>
Message-ID: <10ce01bf049c$169472e0$a602550a@amullhau>
> 1) "[Numeric] C code is a mess." (David Asher)
>
> Could David, Paul or someone elaborate a bit on what this means?
Have a look at the source. If you don't shudder, you've either been
programming not long enough or way too long.
Later,
Andrew Mullhaupt
From mhagger@blizzard.harvard.edu Fri Sep 24 01:52:31 1999
From: mhagger@blizzard.harvard.edu (Michael Haggerty)
Date: 24 Sep 1999 00:52:31 -0000
Subject: [Matrix-SIG] ANNOUNCE: Gnuplot.py 1.3
Message-ID: <19990924005231.13350.qmail@cyclone.harvard.edu>
This is to announce the release of version 1.3 of Gnuplot.py.
Gnuplot.py is a Python [1] package that allows you to create graphs
from within Python using the gnuplot [2] plotting program. This
version fixes plotting of function objects and moves to a package
structure.
Gnuplot.py can be obtained from
http://monsoon.harvard.edu/~mhagger/Gnuplot/Gnuplot.html
Prerequisites (see footnotes):
the Python interpreter [1]
the Python Numeric module [3]
the gnuplot program [2]
Some ways this package can be used:
1. Interactive data processing: Use Python's excellent Numeric package
to create and manipulate arrays of numbers, and use Gnuplot.py to
visualize the results.
2. Web graphics: write CGI scripts in Python that use gnuplot to
output plots in GIF format and return them to the client.
3. Glue for numerical applications (this is my favorite): wrap your
C++/C/Fortran subroutines so that they are callable from Python,
then you can perform numerical computations interactively from
scripts or from the command line and use Gnuplot.py to plot the
output on the fly.
4. Compute a series of datasets in Python and plot them one after the
other using Gnuplot.py to produce a crude animation.
Features added in version 1.3:
* Converted to package format. The main file is now called
__init__.py, which can be loaded by typing 'import Gnuplot'.
* Passing GridData a callable function was basically broken because of
the kludgey way of overloading the argument. Instead of trying to
fix it, I moved that functionality to a new type of PlotItem called
'GridFunc'.
Features already present in older versions:
+ Two and three-dimensional plots.
+ Plot data from memory, from a file, or from an expression.
+ Support for multiple simultaneous gnuplot sessions.
+ Can pass arbitrary commands to the gnuplot program.
+ Object oriented, extensible design with several built-in types
of plot items.
+ Portable and easy to install (nothing to compile except on
Windows).
+ Support for MS Windows, using the `pgnuplot.exe' program.
+ Support for sending data to gnuplot as `inline' or `binary' data.
These are optimizations that also remove the need for temporary
files. Temporary files are also still supported.
Footnotes:
----------
[1] Python <http://www.python.org> is an excellent object-oriented
scripting/rapid development language that is also especially good
at gluing programs together.
[2] gnuplot <ftp://ftp.gnuplot.vt.edu/pub/gnuplot/> is a free,
popular, very portable plotting program with a command-line
interface. It can make 2-d and 3-d plots and can output to myriad
printers and graphics terminals.
[3] The Numeric Python extension
<ftp://ftp-icf.llnl.gov/pub/python/README.html> is a Python module
that adds fast and convenient array manipulations to the Python
language.
Please pass along any comments, suggestions, or complaints!
Yours,
Michael
--
Michael Haggerty
mhagger@blizzard.harvard.edu
From ryszard@arqule.com Fri Sep 24 12:42:02 1999
From: ryszard@arqule.com (Czerminski, Ryszard)
Date: Fri, 24 Sep 1999 07:42:02 -0400
Subject: [Matrix-SIG] ANNOUNCE: Gnuplot.py 1.3
Message-ID: <6B00FB949906D211A88800104B8AE520F419F7@WIREHEAD>
I am looking for a function which could compute
combinations given lexicographical index e.g.
C(1;3,2) -> 1,2
C(2;3,2) -> 1,3
C(3;3,2) -> 2,3
I have found function which does this
( http://math.nist.gov/GAMS.html ) but it is limited
to small numbers since it is using regular 4 bytes
representation for integers and therefore index
range is severly limited ( < 2^32 ).
Any pointers to the software which does this for
integers of arbitrary length would be very much
appreciated.
Ryszard Czerminski phone: (781)395-1269 x 479
ArQule, Inc. e-mail: ryszard@arqule.com
200 Boston Avenue http://www.arqule.com
Medford, MA 02155
From p.s.craig@durham.ac.uk Fri Sep 24 16:49:57 1999
From: p.s.craig@durham.ac.uk (Peter Craig)
Date: Fri, 24 Sep 1999 16:49:57 +0100
Subject: [Matrix-SIG] Bug(s) in Numeric
Message-ID: <37EB9DA5.7F0FDC04@durham.ac.uk>
This is a multi-part message in MIME format.
--------------67510CF3F1534029D986A5E6
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Greetings all,
I have finally found two bugs in the Numeric extension which for some
time have been causing occasional segmentation faults in a fairly
heavily used piece of software in my department. I attach a patch for
arrayobject.c from LLNLDistribution11 which fixes both bugs.
Note that these bugs affect only programs using the facility to process
arrays of arbitrary python objects. Arrays of integers, floating point
and strings are not affected.
The bug in both cases is that a pointer from malloc is being incremented
before the call to free!
Cheers,
Peter Craig
--------------67510CF3F1534029D986A5E6
Content-Type: text/plain; charset=us-ascii;
name="patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="patch"
--- arrayobject.c.dist Fri Apr 2 01:14:45 1999
+++ arrayobject.c Fri Sep 24 16:15:28 1999
@@ -159,7 +159,7 @@
/* every python object in the array. */
extern int PyArray_INCREF(PyArrayObject *mp) {
int i, n;
- PyObject **data;
+ PyObject **data, **data2;
if (mp->descr->type_num != PyArray_OBJECT) return 0;
@@ -170,16 +170,17 @@
}
n = SIZE(mp);
+ data2 = data;
for(i=0; i<n; i++, data++) Py_XINCREF(*data);
- if (!ISCONTIGUOUS(mp)) free(data);
+ if (!ISCONTIGUOUS(mp)) free(data2);
return 0;
}
extern int PyArray_XDECREF(PyArrayObject *mp) {
int i, n;
- PyObject **data;
+ PyObject **data, **data2;
if (mp->descr->type_num != PyArray_OBJECT) return 0;
@@ -190,9 +191,10 @@
}
n = SIZE(mp);
+ data2 = data;
for(i=0; i<n; i++, data++) Py_XDECREF(*data);
- if (!ISCONTIGUOUS(mp)) free(data);
+ if (!ISCONTIGUOUS(mp)) free(data2);
return 0;
}
--------------67510CF3F1534029D986A5E6--
From tim_one@email.msn.com Sat Sep 25 08:17:41 1999
From: tim_one@email.msn.com (Tim Peters)
Date: Sat, 25 Sep 1999 03:17:41 -0400
Subject: [Matrix-SIG] ANNOUNCE: Gnuplot.py 1.3
In-Reply-To: <6B00FB949906D211A88800104B8AE520F419F7@WIREHEAD>
Message-ID: <000001bf0726$0ea96f40$582d153f@tim>
[Czerminski, Ryszard]
> I am looking for a function which could compute
> combinations given lexicographical index e.g.
>
> C(1;3,2) -> 1,2
> C(2;3,2) -> 1,3
> C(3;3,2) -> 2,3
>
> I have found function which does this
> ( http://math.nist.gov/GAMS.html ) but it is limited
> to small numbers since it is using regular 4 bytes
> representation for integers and therefore index
> range is severly limited ( < 2^32 ).
As is often the case, relaxing such limits creates other problems; in this
case, the Fortran TOMS 515 algorithm recomputes the number of combinations
from scratch each time thru the inner loop, which is very expensive if
choosing many elements from a very large set.
> Any pointers to the software which does this for
> integers of arbitrary length would be very much
> appreciated.
One is attached. Note that since this is Python instead of Fortran,
everything is 0-based. That is, when asking for the i'th combination of m
things taken n at a time, i ranges from 0 up to but not including C(m, n),
and the "canonical set" is taken to be range(m). The inner loop avoids the
problem above, so this is reasonably fast even for multi-hundred digit
indices.
more-combinations-than-a-reasonable-person-could-want<wink>-ly y'rs - tim
def _chop(n):
"""n -> int if it fits, else long."""
try:
return int(n)
except OverflowError:
return n
def comb(m, n):
"""m, n -> number of combinations of m items, n at a time.
m >= n >= 0 required.
"""
if not m >= n >= 0:
raise ValueError("m >= n >= 0 required: " + `m, n`)
if n > (m >> 1):
n = m-n
if n == 0:
return 1
result = long(m)
i = 2
m, n = m-1, n-1
while n:
# assert (result * m) % i == 0
result = result * m / i
i = i+1
n = n-1
m = m-1
return _chop(result)
def combatindex(m, n, i):
"""m, n, i -> i'th combination of m items taken n at a time.
m >= n >= 1 and 0 <= i < comb(m, n) required.
Return the i'th combination in lexicographic order, as a list
of n elements taken from range(m).
The index (i) is 0-based.
Example:
>>> for i in range(6):
... print combatindex(4, 2, i)
[0, 1]
[0, 2]
[0, 3]
[1, 2]
[1, 3]
[2, 3]
"""
if not m >= n >= 1:
raise ValueError("m >= n >= 1 required: " + `m, n`)
c = long(comb(m, n))
if not 0 <= i < c:
raise ValueError("0 <= i < comb(m,n) required: " + `i, c`)
result = []
# have c == comb(m, n), want comb(m-1,n-1)
c = c * n / m
# invariant: c == comb(m-1, n-1)
for element in xrange(m):
if i < c:
# take this element, and n-1 from the remaining
result.append(element)
n = n-1
if n == 0:
break
# have c == comb(m-1,n), want comb(m-2,n-1)
c = c * n / (m-1)
else:
# skip this element, and take all from the remaining
i = i-c
# have c == comb(m-1,n-1), want comb(m-2,n-1)
c = c * (m-n) / (m-1)
m = m-1
assert i == 0
return result
def _test():
failures = 0
m, n = 10, 6
c = comb(m, n)
last = [0] * n
for i in xrange(c):
this = combatindex(m, n, i)
if len(this) != n:
print "*** length error:", m, n, i, this
failures = failures + 1
if not last < this:
print "*** lexicographic error:", m, n, i, last, this
failures = failures + 1
last = this
if this != range(m-n, m):
print "*** last value wrong:", m, n, c-1, this
failures = failures + 1
# c has more than 1400 digits in the next case -- may take a
# few seconds
m, n = 5000, 2000
c = comb(m, n)
this = combatindex(m, n, 0)
if this != range(n):
print "*** first value wrong:", m, n, 0, this
failures = failures + 1
this = combatindex(m, n, c-1)
if this != range(m-n, m):
print "*** last value wrong:", m, n, c-1, this
failures = failures + 1
if failures:
print "*********** TEST FAILED *************"
if __name__ == "__main__":
_test()
From ryszard@arqule.com Wed Sep 29 19:08:07 1999
From: ryszard@arqule.com (Czerminski, Ryszard)
Date: Wed, 29 Sep 1999 14:08:07 -0400
Subject: [Matrix-SIG] Re: combinations (summary)
Message-ID: <6B00FB949906D211A88800104B8AE520F41A0E@WIREHEAD>
This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.
------_=_NextPart_000_01BF0AA5.94F84666
Content-Type: text/plain;
charset="iso-8859-1"
I would like to express my appreciation to the people
who took time to help me with combinations problem:
Tim Peters - for Python code which does EXACTLY what I wanted,
for advise and for pointers to GNU's GMP package
Mirek Gorski - for pointer to GNU's GMP package
http://www.math.iastate.edu/cbergman/crypto/gmp/gmp_toc.html
Pat Walters - for C++ class for generating combinations
Emili Besalu - for pointer to his web page dealing with similar
problems http://iqc.udg.es/~emili/emili_nc.htm
Thank you all very much! -- Ryszard
ORIGINAL QUESTION:
I am looking for a function which could compute
combinations given lexicographical index e.g.
C(1;3,2) -> 1,2
C(2;3,2) -> 1,3
C(3;3,2) -> 2,3
I have found function which does this
( http://math.nist.gov/GAMS.html ) but it is limited
to small numbers since it is using regular 4 bytes
representation for integers and therefore index
range is severly limited ( < 2^32 ).
Any pointers to the software which does this for
integers of arbitrary length would be very much
appreciated.
Ryszard Czerminski phone: (781)395-1269 x 479
ArQule, Inc. e-mail: ryszard@arqule.com
200 Boston Avenue http://www.arqule.com
Medford, MA 02155
P.S.
from Tim Peters:
> I am looking for a function which could compute
> combinations given lexicographical index e.g.
>
> C(1;3,2) -> 1,2
> C(2;3,2) -> 1,3
> C(3;3,2) -> 2,3
>
> I have found function which does this
> ( http://math.nist.gov/GAMS.html ) but it is limited
> to small numbers since it is using regular 4 bytes
> representation for integers and therefore index
> range is severly limited ( < 2^32 ).
As is often the case, relaxing such limits creates other problems; in this
case, the Fortran TOMS 515 algorithm recomputes the number of combinations
from scratch each time thru the inner loop, which is very expensive if
choosing many elements from a very large set.
> Any pointers to the software which does this for
> integers of arbitrary length would be very much
> appreciated.
One is attached. Note that since this is Python instead of Fortran,
everything is 0-based. That is, when asking for the i'th combination of m
things taken n at a time, i ranges from 0 up to but not including C(m, n),
and the "canonical set" is taken to be range(m). The inner loop avoids the
problem above, so this is reasonably fast even for multi-hundred digit
indices.
more-combinations-than-a-reasonable-person-could-want<wink>-ly y'rs - tim
def _chop(n):
"""n -> int if it fits, else long."""
try:
return int(n)
except OverflowError:
return n
def comb(m, n):
"""m, n -> number of combinations of m items, n at a time.
m >= n >= 0 required.
"""
if not m >= n >= 0:
raise ValueError("m >= n >= 0 required: " + `m, n`)
if n > (m >> 1):
n = m-n
if n == 0:
return 1
result = long(m)
i = 2
m, n = m-1, n-1
while n:
# assert (result * m) % i == 0
result = result * m / i
i = i+1
n = n-1
m = m-1
return _chop(result)
def combatindex(m, n, i):
"""m, n, i -> i'th combination of m items taken n at a time.
m >= n >= 1 and 0 <= i < comb(m, n) required.
Return the i'th combination in lexicographic order, as a list
of n elements taken from range(m).
The index (i) is 0-based.
Example:
>>> for i in range(6):
... print combatindex(4, 2, i)
[0, 1]
[0, 2]
[0, 3]
[1, 2]
[1, 3]
[2, 3]
"""
if not m >= n >= 1:
raise ValueError("m >= n >= 1 required: " + `m, n`)
c = long(comb(m, n))
if not 0 <= i < c:
raise ValueError("0 <= i < comb(m,n) required: " + `i, c`)
result = []
# have c == comb(m, n), want comb(m-1,n-1)
c = c * n / m
# invariant: c == comb(m-1, n-1)
for element in xrange(m):
if i < c:
# take this element, and n-1 from the remaining
result.append(element)
n = n-1
if n == 0:
break
# have c == comb(m-1,n), want comb(m-2,n-1)
c = c * n / (m-1)
else:
# skip this element, and take all from the remaining
i = i-c
# have c == comb(m-1,n-1), want comb(m-2,n-1)
c = c * (m-n) / (m-1)
m = m-1
assert i == 0
return result
def _test():
failures = 0
m, n = 10, 6
c = comb(m, n)
last = [0] * n
for i in xrange(c):
this = combatindex(m, n, i)
if len(this) != n:
print "*** length error:", m, n, i, this
failures = failures + 1
if not last < this:
print "*** lexicographic error:", m, n, i, last, this
failures = failures + 1
last = this
if this != range(m-n, m):
print "*** last value wrong:", m, n, c-1, this
failures = failures + 1
# c has more than 1400 digits in the next case -- may take a
# few seconds
m, n = 5000, 2000
c = comb(m, n)
this = combatindex(m, n, 0)
if this != range(n):
print "*** first value wrong:", m, n, 0, this
failures = failures + 1
this = combatindex(m, n, c-1)
if this != range(m-n, m):
print "*** last value wrong:", m, n, c-1, this
failures = failures + 1
if failures:
print "*********** TEST FAILED *************"
if __name__ == "__main__":
_test()
=========================================================
from Pat Walters:
------_=_NextPart_000_01BF0AA5.94F84666
Content-Type: application/octet-stream;
name="combo.cpp"
Content-Disposition: attachment;
filename="combo.cpp"
#include <vector>
#include <algorithm>
#include <iostream.h>
#include <stdio.h>
using namespace std;
#include "combo.h"
// Combination generator class
// Takes a vector of integers as input and generates
// all combinations
/*
int main(int argc, char *argv[])
{
vector<int> b(3);
b[0] = 2;
b[1] = 3;
b[2] = 2;
combo c(b);
while (!c.done())
{
print_v(c.get_current());
c.next();
}
return(0);
}
Produces
0 0 0
0 0 1
0 1 0
0 1 1
0 2 0
0 2 1
1 0 0
1 0 1
1 1 0
1 1 1
1 2 0
1 2 1
*/
// support function to print a vector
void print_v(const vector <int> &v)
{
for (int i = 0; i < v.size(); i++)
printf("%4d ",v[i]);
printf("\n");
fflush(stdout);
}
// constructor - takes a vector of ints
// each int represents the maximum number of possibilities for
// that position.
combo::combo(const vector <int> &x)
{
enough = false;
high.resize(x.size());
curr.resize(x.size());
fill (curr.begin(),curr.end(),0);
for ( int i = 0; i < x.size(); i++)
high[i] = x[i] - 1;
len = high.size();
limit = len-1;
}
// debuging function to show a combination
void combo::show()
{
for (int i = 0; i < curr.size(); i++)
printf("%4d ",curr[i]);
printf("\n");
fflush(stdout);
}
// generate the next combination
void combo::next()
{
bool raised;
if (curr[limit]<high[limit])
curr[limit]++;
else
{
curr[limit]=0;
limit--;
raised=false;
while (!raised && !enough)
{
if (curr[limit]<high[limit])
{
curr[limit]++;
raised=true;
limit=len;
}
else
if (limit==-1)
enough=true;
else
curr[limit]=0;
limit--;
}
}
}
------_=_NextPart_000_01BF0AA5.94F84666
Content-Type: application/octet-stream;
name="combo.h"
Content-Disposition: attachment;
filename="combo.h"
class combo
{
private :
vector<int> high;
bool enough;
int len;
int limit;
vector <int> curr;
public :
combo(const vector <int> &high);
void next();
vector <int> &get_current() {return(curr);};
bool done() {return(enough);}
void show();
};
void print_v(const vector <int> &v);
------_=_NextPart_000_01BF0AA5.94F84666--
From acornet@emory.edu Wed Sep 29 23:03:30 1999
From: acornet@emory.edu (Ben Cornett)
Date: Wed, 29 Sep 1999 18:03:30 -0400
Subject: [Matrix-SIG] Numerical compile problems
Message-ID: <37F28CB2.F4B3420B@emory.edu>
I feel like this question has probably come up before, but since I
couldn't find anything that seemed pertinent in the archives, I post it
here, hoping someone might lend me a hand. I'm having problems
installing Numerical on an SGI Octane R10000 under Irix6.5.4, using the
7.3 compilers. The compilation seems to go ok, but I get one unhappy
python interpreter (core dump) when I attempt to import Numerical. I've
tried various permutations in the 'Setup' file, both with and without
the sgi blas libraries installed. I can't really think of any other
relevant info at the moment. If anyone can help me, I would greatly
appreciate it.
Best,
Ben
From hinsen@cnrs-orleans.fr Thu Sep 30 16:40:13 1999
From: hinsen@cnrs-orleans.fr (Konrad Hinsen)
Date: Thu, 30 Sep 1999 17:40:13 +0200
Subject: [Matrix-SIG] Numerical compile problems
In-Reply-To: <37F28CB2.F4B3420B@emory.edu> (message from Ben Cornett on Wed,
29 Sep 1999 18:03:30 -0400)
References: <37F28CB2.F4B3420B@emory.edu>
Message-ID: <199909301540.RAA06726@chinon.cnrs-orleans.fr>
> couldn't find anything that seemed pertinent in the archives, I post it
> here, hoping someone might lend me a hand. I'm having problems
> installing Numerical on an SGI Octane R10000 under Irix6.5.4, using the
> 7.3 compilers. The compilation seems to go ok, but I get one unhappy
> python interpreter (core dump) when I attempt to import Numerical. I've
Did you try this under a debugger in order to find out *where* it
crashes? Something else you might try is linking the NumPy C modules
statically; that gives you better checking for missing or wrong
symbols, which might well be the cause.
I have compiled NumPy on an Octane system without any problems a
while ago, but I don't remember the Irix and compiler version numbers.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron | Fax: +33-2.38.63.15.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
-------------------------------------------------------------------------------
From heather@v1.wustl.edu Thu Sep 30 23:24:52 1999
From: heather@v1.wustl.edu (Heather Drury)
Date: Thu, 30 Sep 1999 17:24:52 -0500 (CDT)
Subject: [Matrix-SIG] performance hit in c extensions
Message-ID: <199909302224.RAA04282@v1.wustl.edu>
Hi,
I have sucessfully written wrappers for most of my C code so
I can call the modules directly from python. Unfortunately,
it *appears* that the code executes about 50% slower (I haven't
actually timed it) when called from python versus being
called from the command line. I would not have thought
there would be any performance impact.
Does that seem reasonable?
Thanks,
Heather
--
Heather Drury heather@v1.wustl.edu
Washington University School of Medicine http://v1.wustl.edu
Department of Anatomy & Neurobiology Phone: 314-362-4325
660 S. Euclid, MS 8108 FAX: 314-747-4370
St. Louis, MO 63110-1093