From phil@geog.ubc.ca Fri Nov 1 02:22:24 1996
From: phil@geog.ubc.ca (Phil Austin)
Date: Thu, 31 Oct 1996 18:22:24 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <199610311925.OAA28124@python.org>
References: <9610311758.AA16110@goldilocks.LCS.MIT.EDU>
<199610311925.OAA28124@python.org>
Message-ID: <199611010222.SAA06674@coot.geog.ubc.ca>
>>>>> "GF" == Geoffrey Furnish <furnish@laura.llnl.gov> writes:
GF> The PLplot module supports output to a Tk graphics widget
GF> (among a multitude of other output drivers), which can be
GF> trivially integrated into Python/Tk, and we have that in
GF> operation here at LLNL now. There is also the beginnings of a
GF> Python/Tk "megawidget" for the plframe graphics widget, which
GF> provides a variety of conveniences for scientific plotting,
GF> including saving to a any of a variety of supported graphics
GF> formats (including postscript and ppm), as well as
GF> manipulation of the plot orientation, printing the currently
GF> visible view. etc. There is also a binding available for
GF> zoom support, and "zoom frame list walking", with panning.
This is great news. One of my own projects will be positioning Python
as a Matlab replacement for the 7000 undergrads in our faculty of
science. Being able to do graphics under Windows95 is a major
prerequisite for this.
My other hope is that the door is still open on the future addtion of
Splus-style syntactic sugar allowing subscripts that are array masks.
One sure way to impress a new user in Splus is to show them how to
histogram part of a 3-d array with:
hist(array[array > 40. & array < 50.])
I can also commit to providing on-going documentation and support for
the netcdf module in conjunction with Bill Noon.
Phil Austin INTERNET: phil@geog.ubc.ca
(604) 822-2175 FAX: (604) 822-6150
http://www.geog.ubc.ca/~phil
Associate Professor
Atmospheric Sciences Programme
Geography #217
University of British Columbia
1984 W Mall
Vancouver, BC V6T 1Z2
CANADA
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From furnish@laura.llnl.gov Fri Nov 1 03:43:45 1996
From: furnish@laura.llnl.gov (Geoffrey Furnish)
Date: Thu, 31 Oct 1996 19:43:45 -0800
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <199611010222.SAA06674@coot.geog.ubc.ca>
References: <9610311758.AA16110@goldilocks.LCS.MIT.EDU>
<199610311925.OAA28124@python.org>
<199611010222.SAA06674@coot.geog.ubc.ca>
Message-ID: <199611010345.WAA29560@python.org>
Phil Austin writes:
> >>>>> "GF" == Geoffrey Furnish <furnish@laura.llnl.gov> writes:
> GF> The PLplot module supports output to a Tk graphics widget
> GF> (among a multitude of other output drivers), which can be
> GF> trivially integrated into Python/Tk, and we have that in
> GF> operation here at LLNL now. There is also the beginnings of a
> GF> Python/Tk "megawidget" for the plframe graphics widget, which
> GF> provides a variety of conveniences for scientific plotting,
> GF> including saving to a any of a variety of supported graphics
> GF> formats (including postscript and ppm), as well as
> GF> manipulation of the plot orientation, printing the currently
> GF> visible view. etc. There is also a binding available for
> GF> zoom support, and "zoom frame list walking", with panning.
>
>
> This is great news. One of my own projects will be positioning Python
> as a Matlab replacement for the 7000 undergrads in our faculty of
> science. Being able to do graphics under Windows95 is a major
> prerequisite for this.
I will have to back pedal a little. The Tk plframe widget currently
uses the X11 drawing widget, which is not available on
other-than-unix. There has, however, been a patch submitted for a
driver implemented only in terms of Tk library features, which is
cross platform. This widget has not been fully integrated into PLplot
yet, however, and although I can say "it will happen", I cannot be
precise about timing.
--
Geoffrey Furnish email: furnish@llnl.gov
LLNL X/ICF phone: 510-424-4227 fax: 510-423-6172
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From fredrik_lundh@ivab.se Fri Nov 1 08:17:30 1996
From: fredrik_lundh@ivab.se (Fredrik Lundh)
Date: Fri, 1 Nov 1996 09:17:30 +0100
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <Pine.SUN.3.91.961031151427.20286A-100000@lems61>
(pas@lems.brown.edu)
Message-ID: <9611010817.AA09684@arnold.image.ivab.se>
> Sounds good, especially 3. I think light weight Py/Tk widgets would be
> the way to go.
FWIW, PIL 0.2final will include a "Path" object (2D polyline/beziers,
and some path processing primitives), and a first stab at a light-
weight plotting package based on these primitives.
To avoid some of the worst performance bottlenecks in Tk, output
rendering is currently done via raster images (or directly to
PostScript). Future versions will probably include patches allowing
you to place a path directly on a canvas.
Now, could I just find some decent contouring code somewhere...
Regards /F
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Fri Nov 1 14:44:03 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Fri, 1 Nov 96 15:44:03 +0100
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <Pine.SUN.3.91.961031124702.24582C-100000@physics20.Berkeley.EDU>
(message from Johann Hibschman on Thu, 31 Oct 1996 12:47:57 -0800
(PST))
Message-ID: <199611011544.QAA19758@ibs.ibs.fr>
> Meanwhile, I'm keeping myself busy fending off periodic assaults from
> friends saying that I should just use either IDL or C and be done with it.
> My CS friends think perl is the answer to all problems. IDL's great, but
If you want yet another opinion: use Python and be done with it ;-)
Seriously, all such suggestions make sense only for a specific set of
problems to be solved. There is no one and only tool that will make
everyone happy. Python + NumPy + the occasional module written in C
have proven to be the best solution for my needs, but your mileage
may vary.
> would take a zillion lines of code to do anything interesting. I'm just
> not understood. Sigh. :)
You should consider yourself lucky. I have to fight against people who
think that *Fortran* is the sole path to hapiness! ;-)
> (I am somewhat disturbed by reports that perl is several times faster
> than python, though.)
When seeing such a report, your immediate question should be: for what?
Languages like Python and Perl tend to be fast for the purposes that
are supported by special low-level modules (written in C), but can be
slow for anything else. For array-based number crunching, for example,
Python's overall speed is almost irrelevant; all that counts is the
speed of the NumPy extension.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Fri Nov 1 14:45:51 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Fri, 1 Nov 96 15:45:51 +0100
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <Pine.SUN.3.91.961031151427.20286A-100000@lems61>
(pas@lems.brown.edu)
Message-ID: <199611011546.QAA19762@ibs.ibs.fr>
> PLplot is a bear to compile. I've had a fair amount of experience working
> with large software projects, and getting PLplot to the point where I
> could use it, in or out of python, took a significant amount of work. Way
> too much, in my opinion. And it even uses gnu configure.
I have to agree. I tried PLplot a few months ago, but it took forever
just to get everything compiled, and the prospect of having to
do this repeatedly (new machines, new versions) made be prefer to
live without it.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Fri Nov 1 14:51:09 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Fri, 1 Nov 96 15:51:09 +0100
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <199611010222.SAA06674@coot.geog.ubc.ca> (message from Phil
Austin on Thu, 31 Oct 1996 18:22:24 -0800 (PST))
Message-ID: <199611011551.QAA19768@ibs.ibs.fr>
> My other hope is that the door is still open on the future addtion of
> Splus-style syntactic sugar allowing subscripts that are array masks.
> One sure way to impress a new user in Splus is to show them how to
> histogram part of a 3-d array with:
>
> hist(array[array > 40. & array < 50.])
What is that supposed to mean for a 3-d array? Or even for 2-d?
Anyway, adding syntactic sugar like this can even be done by writing a
Python wrapper for arrays, with no changes to NumPy. Even if
that may be too inefficient for real applications, it is a great way
to test new proposals such as yours.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From phil@geog.ubc.ca Fri Nov 1 15:42:44 1996
From: phil@geog.ubc.ca (Phil Austin)
Date: Fri, 1 Nov 1996 07:42:44 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <199611011551.QAA19768@ibs.ibs.fr>
References: <199611010222.SAA06674@coot.geog.ubc.ca>
<199611011551.QAA19768@ibs.ibs.fr>
Message-ID: <199611011542.HAA07159@coot.geog.ubc.ca>
>> hist(array[array > 40. & array < 50.])
>>>>> "KH" == Konrad Hinsen <hinsen@ibs.ibs.fr> writes:
KH> What is that supposed to mean for a 3-d array? Or even for
KH> 2-d?
The default in Splus is to flatten the comparison when
it appears as the sole subscript. If you want to work with individual
indexes, you'd add empty indexes as in python (although there're
no pseudo indexes in splus).
I wonder if anyone else on the list is familiar with Splus besides
David Ascher? I've used it since 1982 (it was the instructional
language for Washington's grad stats courses), and its compact
indexing notation is unambiguous and very intuitive (to me). Perhaps
I'll implement some of its features as Konrad suggests (possibly as
soon as my 2 year-old starts elementary school).
Phil
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From da@maigret.cog.brown.edu Fri Nov 1 16:22:52 1996
From: da@maigret.cog.brown.edu (David Ascher)
Date: Fri, 1 Nov 1996 11:22:52 -0500 (EST)
Subject: [PYTHON MATRIX-SIG] Calling all extensions...
Message-ID: <199611011622.LAA16382@maigret>
I'll be giving a talk at the Python Workshop on Monday. One of the
things I'll be spending a couple of minutes on because Jim can't go is a
status report of the Numeric extensions. If you'd like me to mention that
you're working on a specific extension, please let me know ASAP. The
ones I have on my overhead as of now are: LAPACK, FFTPACK, RANLIB,
NetCDF, PLPLOT, OpenGL, GIST (although the Gist extension is the subject
of its own presentation by its authors) and Konrad's extensions.
Anything else that should be mentioned?
--david
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From furnish@laura.llnl.gov Fri Nov 1 17:56:33 1996
From: furnish@laura.llnl.gov (Geoffrey Furnish)
Date: Fri, 1 Nov 1996 09:56:33 -0800
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <199611011546.QAA19762@ibs.ibs.fr>
References: <Pine.SUN.3.91.961031151427.20286A-100000@lems61>
<199611011546.QAA19762@ibs.ibs.fr>
Message-ID: <199611011758.MAA01364@python.org>
Konrad Hinsen writes:
> > PLplot is a bear to compile. I've had a fair amount of experience working
> > with large software projects, and getting PLplot to the point where I
> > could use it, in or out of python, took a significant amount of work. Way
> > too much, in my opinion. And it even uses gnu configure.
>
> I have to agree. I tried PLplot a few months ago, but it took forever
> just to get everything compiled, and the prospect of having to
> do this repeatedly (new machines, new versions) made be prefer to
> live without it.
With all this public bashing of the PLplot configuration support, I
think I have to point out some facts which will be missed by people
who have never actually tried it and are going to be very disuaded by
statements like this, which I personally regard as fairly
exaggerated. Here's why:
First, it is certainly unarguable that PLplot's configuration script
(GNU autoconf based), has historically had a lot of trouble correctly
identifying every single part of a user's supporting environment
during autoconfiguration. And I won't deny the frustration many have
felt, including the two quoted above.
However, you have to understand clearly that the /worst possible case/
with PLplot is exactly the /standard/ case with Python and NumPy
itself. In particular, PLplot's configure works very hard to
automatically detect the presence and version number of all the
supporting libraries the user may have. X11, Tcl/Tk, Python, and
others, all fall into this category. And it is true, I admit, that
PLplot's configure has, on some systems, had trouble sorting it all
out.
Note, however, that both Python and NumPy, make absolutely no effort
whatsoever to "auto detect" this kind of stuff. For instance, say you
are building python on a system on which you have already installed
Tcl/Tk. Is Tkinter automatically configured? No. No attempt
whatsoever is made to find your Tcl/Tk, and edit your Setup to turn on
the compilation of the Tkinter module. You have to do it by hand,
editing Setup, changing the lines for the tkinter module, filling in
correct path information, etc. You have to edit _tkinter.c by hand to
patch in package initializations which Python doesn't know about, etc.
When you install NumPy, does it look to see if you have a precompiled
FFTPACK sitting around? No. The instructions suggest you use the C
version provided, and if you want to use some other version, you have
to edit the setup by hand. No big deal. Nobody complains about this,
and neither do I.
And that situation, is the worst possible situation with PLplot. If
autoconf doesn't do all the magic for you, you get to do it by hand.
If you try to compile PLplot against Tcl 6.7 and Tk 3.6, and you have
trouble, the situation is no different than if you did the same
blunder with Python's Tkinter. Etc.
What PLplot does do, that neither Python nor NumPy do, is try very
hard to sort out what you have on your system, and autoconfigure
everything possible. For large numbers of people (read, thouusands)
this has worked adequately. For some, it has failed miserably. For
those unlucky souls, the worst thing that happens is they have to make
a barebones makefile, and edit it by hand, which is exactly what
/everyone/ has to do with Python and NumPy themselves.
And, moreover, I have recently corrected dozens of autodetection bugs
in PLplot's configure, so that the number of "unlucky souls" will be
substantially lower than it was before.
Anyone who /still/ has major problems with the PLplot autoconf
situation is respectfully invited to air their grievances on the
PLplot mailing list, where there are plenty of people who could help
them with valuable advice.
--
Geoffrey Furnish email: furnish@llnl.gov
LLNL X/ICF phone: 510-424-4227 fax: 510-423-6172
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From drh@cse.unl.edu Fri Nov 1 18:31:32 1996
From: drh@cse.unl.edu (Doug Heisterkamp)
Date: Fri, 1 Nov 1996 12:31:32 -0600 (CST)
Subject: [PYTHON MATRIX-SIG] Calling all extensions...
Message-ID: <199611011831.MAA16560@cse.unl.edu>
>
> I'll be giving a talk at the Python Workshop on Monday. One of the
> things I'll be spending a couple of minutes on because Jim can't go is a
> status report of the Numeric extensions. If you'd like me to mention that
> you're working on a specific extension, please let me know ASAP. The
> ones I have on my overhead as of now are: LAPACK, FFTPACK, RANLIB,
> NetCDF, PLPLOT, OpenGL, GIST (although the Gist extension is the subject
> of its own presentation by its authors) and Konrad's extensions.
> Anything else that should be mentioned?
>
I have a low-level access module to ODRPACK. One thing to note is that I
have it setup so I can use Python, C, or Fortran functions in ODRPACK. I'm
in the process of generalizing this into a separate module so I can use it
with other libraries. I will put it up for ftp soon (probably sometime
next week). ODRPACK itself is available from netlib.
Doug Heisterkamp
drh@cse.unl.edu
PS: A short blurb on ODRPACK from it's readme file:
*** INTRODUCTION
ODRPACK is a portable collection of Fortran subprograms for fitting a model to
data. It is designed primarily for instances when the explanatory as well as
the response variables have significant errors, implementing a highly
efficient algorithm for solving the weighted orthogonal distance regression
problem, i.e., for minimizing the sum of the squares of the weighted
orthogonal distances between each data point and the curve described by the
model equation. It can also be used to solve the ordinary least squares
problem where all of the errors are attributed to the observations of the
dependent variable. File CHANGES identifies how the current version of
ODRPACK differs from earlier releases.
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From tom.schwaller@linux-magazin.de Sat Nov 2 02:04:20 1996
From: tom.schwaller@linux-magazin.de (Tom Schwaller)
Date: Sat, 02 Nov 1996 03:04:20 +0100
Subject: [PYTHON MATRIX-SIG] Calling all extensions...
References: <199611011831.MAA16560@cse.unl.edu>
Message-ID: <327AAC24.4EEAE11E@linux-magazin.de>
Doug Heisterkamp wrote:
> I have a low-level access module to ODRPACK. One thing to note is that I
> have it setup so I can use Python, C, or Fortran functions in ODRPACK. I'm
> in the process of generalizing this into a separate module so I can use it
> with other libraries. I will put it up for ftp soon (probably sometime
> next week). ODRPACK itself is available from netlib.
>
Funny enough, we redo all the work done many times before (Matlab,
Octave, Scilab and
the last in this series
http://zaphod.uni-muenster.de:80/xi/
Your general approach seems interesting. We should work much more in
that direction.
Btw, has anybody written a general module to dynamicaly load numericl
procedures
after declaring their argument types. There is a module in the python
distribution,
but it's very rudimentary.
I do not remeber the exact form in Scilab, but in Python it should work
like:
C-Function: double func(int n, double* a, int r, double s)
f=dynload(func, 'd', 'i', 'D', 'i', 'd') (or f=funcload('diDid') , or
f=funcload('d', 'iDid'))
res = f(10, arange(10), 3, 5.3)
Some months ago I also saw a Tcl-extenion for compiling simple
C-Cyntax like expressions (just Tcl strings) to C-Functions on the fly..
It works like the example above, but instead of having the C-function at
the beginning,
you declare it with something like:
f=declarefunc('d', 'iDid', 'nars', "\
int j=0;
double res=0;
for (j=0; j<n; j++) {
res += a[j];
}
return res*r - s;
}")
res = f(10, arange(10), 3, 5.3))
This is a dirty, not so Python-like solution, but is there a better one
for scientific computations. For heavy computations you need compiled
functions, which you can
generate on the fly, evaluating a interpreted function 10000 time for a
computation
is to slow. Any comments on that.
What about sparse matrices. I remeber we had a discussion on that a very
long time ago.
I propose taking the sparsepack library and melting it with NumPy.
Tom
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From templon@studbolt.mit.edu Sat Nov 2 02:55:49 1996
From: templon@studbolt.mit.edu (Jeffrey Templon)
Date: Fri, 1 Nov 1996 21:55:49 -0500
Subject: [PYTHON MATRIX-SIG] errors and questions in building latest release
Message-ID: <199611020255.VAA17969@studbolt.mit.edu>
Hi,
When building the latest release (Linux 2.0.20, gcc 2.7.0), I get lots
of the following errors when compiling NumPy
./multiarraymodule.c:1003: warning: comparison between pointer and integer
./multiarraymodule.c: In function `array_binarysearch':
I mean to say lots of lines in multiarray give the same error.
Also, when does a NumPy arrive which compiles with python 1.4?
One more -- has anybody written any (fairly flexible)
nonlinear-least-squares package (esp. where the proper bookkeeping is
done so that one can selectively fix parameters durin fit passes)?
JT
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From da@maigret.cog.brown.edu Sat Nov 2 15:55:34 1996
From: da@maigret.cog.brown.edu (David Ascher)
Date: Sat, 2 Nov 1996 10:55:34 -0500 (EST)
Subject: [PYTHON MATRIX-SIG] errors and questions in building latest release
In-Reply-To: <199611020255.VAA17969@studbolt.mit.edu> from "Jeffrey Templon" at Nov 1, 96 09:55:49 pm
Message-ID: <199611021555.KAA17392@maigret>
> ./multiarraymodule.c:1003: warning: comparison between pointer and integer
> ./multiarraymodule.c: In function `array_binarysearch':
>
> I mean to say lots of lines in multiarray give the same error.
Note that these are warnings, not errors == it'll work just fine.
> Also, when does a NumPy arrive which compiles with python 1.4?
Don't know, but FYI, to get the latest a5 to compile with 1.4, all that
is needed is changing Ellipses to Ellipsis in arrayobject.c. That
doesn't fix the other bugs in 1.0a5, but at least it works.
--david
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From jbaddor@phy.ulaval.ca Sun Nov 3 15:52:27 1996
From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR)
Date: Sun, 3 Nov 1996 10:52:27 -0500 (EST)
Subject: [PYTHON MATRIX-SIG] LinAlg.py vs LinearAlgebra.py
Message-ID: <Pine.SUN.3.95.961103104843.22970A-100000@mills>
Hi,
Does the new LinearAlgebra.py package require the matrix to be transposed
before and after treatment, like the old Linalg ?
Jean-Bernard
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Sun Nov 3 16:45:06 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Sun, 3 Nov 96 17:45:06 +0100
Subject: [PYTHON MATRIX-SIG] Data analysis package question
In-Reply-To: <199611011858.TAA20155@ibs.ibs.fr> (message from Geoffrey Furnish
on Fri, 1 Nov 1996 09:56:33 -0800)
Message-ID: <199611031745.SAA23261@ibs.ibs.fr>
> However, you have to understand clearly that the /worst possible case/
> with PLplot is exactly the /standard/ case with Python and NumPy
> itself. In particular, PLplot's configure works very hard to
> automatically detect the presence and version number of all the
> supporting libraries the user may have. X11, Tcl/Tk, Python, and
> others, all fall into this category. And it is true, I admit, that
> PLplot's configure has, on some systems, had trouble sorting it all
> out.
Well, I prefer to be told to do it by hand rather than have to figure
out from an unclear error message that some program was trying to be
smart and failed. Besides, it wasn't just this kind of problem. PLplot
didn't find one of its own header files, and I even had to modify some
of the source files because they wouldn't get along with the SGI
system headers.
> Note, however, that both Python and NumPy, make absolutely no effort
> whatsoever to "auto detect" this kind of stuff. For instance, say you
> are building python on a system on which you have already installed
> Tcl/Tk. Is Tkinter automatically configured? No. No attempt
> whatsoever is made to find your Tcl/Tk, and edit your Setup to turn on
> the compilation of the Tkinter module. You have to do it by hand,
With Python, I follow a series of documented steps and get a working
installation within minutes. With PLplot, it took me two days before
I could run the test examples (of course I didn't work full time on it).
That's the bottom line.
> And that situation, is the worst possible situation with PLplot. If
> autoconf doesn't do all the magic for you, you get to do it by hand.
Fine, but that ought to be pointed out clearly.
> Anyone who /still/ has major problems with the PLplot autoconf
> situation is respectfully invited to air their grievances on the
> PLplot mailing list, where there are plenty of people who could help
> them with valuable advice.
Well, I'll try the latest version as soon as I find some free minutes...
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Sun Nov 3 17:09:35 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Sun, 3 Nov 96 18:09:35 +0100
Subject: [PYTHON MATRIX-SIG] LinAlg.py vs LinearAlgebra.py
In-Reply-To: <Pine.SUN.3.95.961103104843.22970A-100000@mills> (message from
Jean-Bernard ADDOR on Sun, 3 Nov 1996 10:52:27 -0500 (EST))
Message-ID: <199611031810.TAA23321@ibs.ibs.fr>
> Does the new LinearAlgebra.py package require the matrix to be transposed
> before and after treatment, like the old Linalg ?
>From the point of view of the user, neither the old nor the new
package should require any transposition. And since internally they
both use LAPACK, there shouldn't be any difference either.
If you ever transposed matrices while working with LinearAlgebra,
I suppose your interpretation of matrix indices is different from
the one adopted in NumPy!
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Sun Nov 3 17:30:36 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Sun, 3 Nov 96 18:30:36 +0100
Subject: [PYTHON MATRIX-SIG] Calling all extensions...
In-Reply-To: <327AAC24.4EEAE11E@linux-magazin.de> (message from Tom Schwaller
on Sat, 02 Nov 1996 03:04:20 +0100)
Message-ID: <199611031831.TAA23361@ibs.ibs.fr>
> This is a dirty, not so Python-like solution, but is there a better one
> for scientific computations. For heavy computations you need compiled
> functions, which you can
> generate on the fly, evaluating a interpreted function 10000 time for a
> computation
> is to slow. Any comments on that.
Good question. A while ago I tried the approach of implementing a
data type "numerical expression" that would pretend to be a numerical
data type and could therefore be passed to any Python code doing
numerical calculations. The result would be a compiled version of that
code (still using a bytecode interpreter, but with typed data objects
and no memory allocation overhead during evaluation) that would be
a callable Python object.
It never got finished, partly due to time restrictions. However, I
realised that this approach had a few problems:
1) The speedup would be significant only for complicated code or
loops, but since Python's branching and looping constructs
could not be used, the idea of profitably compiling ordinary
Python code in this way is probably illusionary.
2) Python's implementation of comparisons made using this extension
really messy for nontrivial application.
3) For really CPU intensive calculations the stack-based evaluation
engine was still too slow.
Nevertheless, I still think that some compilation method not requiring
to run the C compiler would be nice as an intermediate step. Not
everyone knows C, and not all systems support dynamic libraries, without
which the idea of just-in-time compilation doesn't make sense.
On the other hand, an easy way to interface to existing C and Fortran
functions would obviously be nice as way. "Easy" would ideally mean an
approach that can be used by someone who does not know C and/or Fortran.
Unfortunately I don't have any good solution at hand...
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Mon Nov 4 09:34:19 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Mon, 4 Nov 96 10:34:19 +0100
Subject: [PYTHON MATRIX-SIG] C API questions
Message-ID: <199611041034.LAA25034@ibs.ibs.fr>
For the first time I have to use the C API to the array objects in a
non-trivial way, and I'd like to profit a bit from the expertise of
those who have done it before.
I want to provide a C interface to functions written in Python, which
means I have to construct array objects referring to a given C array
of arbitrary origin. This raises a few questions about memory allocation.
and reference counting.
1) How should I construct the array object to make sure that Python
will not try to allocate or free memory for the actual data?
2) Should my code simply pass array objects to the Python code and
then forget about them, or should it increase the reference count
before the call and decrement it afterwards? I suppose that if
everything goes well that shouldn't make a difference.
3) What should I do to prevent the Python code from keeping a reference
to the arrays even after returning (e.g. by putting it into a
global list)? Is there any legitimate reason to do this? Should I
check the reference count after the call and raise an exception if
it is too high? I must somehow make sure that the temporary array
object will be deleted, since I can't guarantee for the existence
of the data part after my function finishes its work.
4) Is there any other problem I should watch out for?
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From drh@cse.unl.edu Mon Nov 4 20:03:37 1996
From: drh@cse.unl.edu (Doug Heisterkamp)
Date: Mon, 4 Nov 1996 14:03:37 -0600 (CST)
Subject: [PYTHON MATRIX-SIG] Calling all extensions...
In-Reply-To: <327AAC24.4EEAE11E@linux-magazin.de> from "Tom Schwaller" at Nov 2, 96 03:04:20 am
Message-ID: <199611042003.OAA19994@cse.unl.edu>
>
>
> Your general approach seems interesting. We should work much more in
> that direction.
>
> Btw, has anybody written a general module to dynamicaly load numericl
> procedures
> after declaring their argument types. There is a module in the python
> distribution,
> but it's very rudimentary.
I assume you mean dlmodule. I have not look at that module before -- I'd
assume it was Sun specific. Is it portable and is there documentation
for it?
>
> I do not remeber the exact form in Scilab, but in Python it should work
> like:
>
> C-Function: double func(int n, double* a, int r, double s)
>
> f=dynload(func, 'd', 'i', 'D', 'i', 'd') (or f=funcload('diDid') , or
> f=funcload('d', 'iDid'))
> res = f(10, arange(10), 3, 5.3)
>
I think something like this is possible. I would prefer separating the
return type from the parameter types and making the return type optional
with a default of returning None. I would also like an optional argument
for the library which holds the function. The default libraries list
could be kept in the dynamic load module.
> Some months ago I also saw a Tcl-extenion for compiling simple
> C-Cyntax like expressions (just Tcl strings) to C-Functions on the fly..
>
<snip>
>
> This is a dirty, not so Python-like solution, but is there a better one
> for scientific computations. For heavy computations you need compiled
> functions, which you can
> generate on the fly, evaluating a interpreted function 10000 time for a
> computation
> is to slow. Any comments on that.
Generating compiled functions on the fly can be done right now if you
have a set interface. Doing it portably could be difficult. But it
can be done by (1) writing the text to a file, (2) executing a makefile
that (a) compiles the text file, (b) places it into a library, (c) relinks
the dynamically loaded module that holds the interface function with
the library, (d) copy the module to python's sharedmodule directory,
(3) reload the module.
If we can get a dynload module working then the restriction on having
a set interface is gone, and also the restriction of using only a
certain number of functions at one time is gone (the interface module
has a static number of function interfaces).
As to interpreted functions being to slow for heavy computations, I think
it will depend on the problem. But in general, I think it will be true.
I'm hopeful that if the function is using the array operations then
it will run in a reasonable amount of time. Even if it does not, there
is always late friday afternoons. Where letting the machine run over the
weekend is much better than staying into the evening and writing the
C functions.
>
> What about sparse matrices. I remeber we had a discussion on that a very
> long time ago.
> I propose taking the sparsepack library and melting it with NumPy.
Which sparse matrix library do you use? I don't need to use sparse
matrices at the moment, but I would be willing to help someone else
in setting up the interface with NumPy.
Doug Heisterkamp
drh@cse.unl.edu
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From drh@cse.unl.edu Mon Nov 4 20:50:52 1996
From: drh@cse.unl.edu (Doug Heisterkamp)
Date: Mon, 4 Nov 1996 14:50:52 -0600 (CST)
Subject: [PYTHON MATRIX-SIG] C API questions
In-Reply-To: <199611041034.LAA25034@ibs.ibs.fr> from "Konrad Hinsen" at Nov 4, 96 10:34:19 am
Message-ID: <199611042050.OAA25684@cse.unl.edu>
> For the first time I have to use the C API to the array objects in a
> non-trivial way, and I'd like to profit a bit from the expertise of
> those who have done it before.
>
> I want to provide a C interface to functions written in Python, which
> means I have to construct array objects referring to a given C array
> of arbitrary origin. This raises a few questions about memory allocation.
> and reference counting.
>
I'm am not an expert on the C API, but I am doing similar things. I'll
relate what I'm currently doing in the hope that I'll be corrected
if it is not right.
> 1) How should I construct the array object to make sure that Python
> will not try to allocate or free memory for the actual data?
I am using PyArray_FromDimsAndData. If data is passed into this function,
then the flag OWN_DATA is not set. In array_dealloc, if OWN_DATA is not
set, then the data pointer of the array object is not freed.
>
> 2) Should my code simply pass array objects to the Python code and
> then forget about them, or should it increase the reference count
> before the call and decrement it afterwards? I suppose that if
> everything goes well that shouldn't make a difference.
I decrement the reference to the array object after the python function
call. I do not increment it before the function call. The array
object reference count was incremented in the PyArray_FromDimsAndData
function by its call to PyObject_NEW (I think--not very sure on some of
this stuff).
>
> 3) What should I do to prevent the Python code from keeping a reference
> to the arrays even after returning (e.g. by putting it into a
> global list)? Is there any legitimate reason to do this? Should I
> check the reference count after the call and raise an exception if
> it is too high? I must somehow make sure that the temporary array
> object will be deleted, since I can't guarantee for the existence
> of the data part after my function finishes its work.
>
I have not done this, but I should. Checking the reference count and
raising an exception should work in most cases.
> 4) Is there any other problem I should watch out for?
Just the typical stuff: remember to DECREF the argument list if you
created it with Py_BuildValue, remember the DECREF the results returned
from the python function when you are done with them, and if calling
from a Fortran function remember to reverse the indexing order when
building the array object.
Doug Heisterkamp
drh@cse.unl.edu
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From tom.schwaller@linux-magazin.de Mon Nov 4 23:59:55 1996
From: tom.schwaller@linux-magazin.de (Tom Schwaller)
Date: Tue, 05 Nov 1996 00:59:55 +0100
Subject: [PYTHON MATRIX-SIG] Calling all extensions and Sparselib
References: <199611042003.OAA19994@cse.unl.edu>
Message-ID: <327E837B.5FFC6C57@linux-magazin.de>
>
> I assume you mean dlmodule. I have not look at that module before -- I'd
> assume it was Sun specific. Is it portable and is there documentation
> for it?
Hmm, the dl interface is implemented on all architectures I know.
When you have dynamic loading of modules running,
then the dlmodule should work too..
> >
> > I do not remeber the exact form in Scilab, but in Python it should work
> > like:
> >
> > C-Function: double func(int n, double* a, int r, double s)
> >
> > f=dynload(func, 'd', 'i', 'D', 'i', 'd') (or f=funcload('diDid') , or
> > f=funcload('d', 'iDid'))
> > res = f(10, arange(10), 3, 5.3)
> >
>
> I think something like this is possible. I would prefer separating the
> return type from the parameter types and making the return type optional
> with a default of returning None. I would also like an optional argument
> for the library which holds the function. The default libraries list
> could be kept in the dynamic load module.
Good point.
>
> Generating compiled functions on the fly can be done right now if you
> have a set interface. Doing it portably could be difficult. But it
> can be done by (1) writing the text to a file, (2) executing a makefile
> that (a) compiles the text file, (b) places it into a library, (c) relinks
> the dynamically loaded module that holds the interface function with
> the library, (d) copy the module to python's sharedmodule directory,
> (3) reload the module.
Thats exactly the approcha of the mentioned Tcl module, whichhhhh works
quite well..
>
> As to interpreted functions being to slow for heavy computations, I think
> it will depend on the problem. But in general, I think it will be true.
> I'm hopeful that if the function is using the array operations then
> it will run in a reasonable amount of time. Even if it does not, there
> is always late friday afternoons. Where letting the machine run over the
> weekend is much better than staying into the evening and writing the
> C functions.
>
I had some tests 2 years ago with PDE stuff where I used
Python to plug in functions (written very fast with Python)
and the results where absolutely disapointing.
Also Tests with nonlinear optimisation stuff were very bad..
> > What about sparse matrices. I remeber we had a discussion on that a very
> > long time ago.
> > I propose taking the sparsepack library and melting it with NumPy.
>
> Which sparse matrix library do you use? I don't need to use sparse
> matrices at the moment, but I would be willing to help someone else
> in setting up the interface with NumPy.
Im not exactly shure which one I one discoverd on the net, but I think
it's
http://math.nist.gov/spblas/
There are also C++ Wrapper code at
http://math.nist.gov/sparselib++
Very important for our purpose is
http://math.nist.gov/acmd/Staff/KRemington/spblasc/spblasc.html
http://www.cs.umn.edu/research/darpa/p_sparslib/psp-abs.html
is also very interesting.
I propose to use the first code (Sparselib) and here"s another link
http://math.nist.gov/iml++
which one should consider for preconditioning stuff..
Hope this is interesting stuff, if you do not know it.
Tom
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From phil@geog.ubc.ca Tue Nov 5 00:59:43 1996
From: phil@geog.ubc.ca (Phil Austin)
Date: Mon, 4 Nov 1996 16:59:43 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] Calling all extensions and Sparselib
In-Reply-To: <327E837B.5FFC6C57@linux-magazin.de>
References: <199611042003.OAA19994@cse.unl.edu>
<327E837B.5FFC6C57@linux-magazin.de>
Message-ID: <199611050059.QAA09421@coot.geog.ubc.ca>
>>>>> "Tom" == Tom Schwaller <tom.schwaller@linux-magazin.de> writes:
>> Which sparse matrix library do you use? I don't need to use sparse
>> matrices at the moment, but I would be willing to help someone else
>> in setting up the interface with NumPy.
Tom> Im not exactly shure which one I one discoverd on the net, but I think
Tom> it's
Tom> http://math.nist.gov/spblas/
Also worth a look is the more elaborate:
<DT><A HREF="http://www.mcs.anl.gov/petsc/petsc.html"
LAST_MODIFIED="847131485">PETSc - The Portable, Extensible Toolkit
for Scientific Computation: sparse pdes</A>
Phil
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Tue Nov 5 17:52:29 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 5 Nov 96 18:52:29 +0100
Subject: [PYTHON MATRIX-SIG] C API questions
In-Reply-To: <199611042048.OAA25223@cse.unl.edu> (drh@cse.unl.edu)
Message-ID: <199611051752.SAA02842@ibs.ibs.fr>
> > 1) How should I construct the array object to make sure that Python
> > will not try to allocate or free memory for the actual data?
>
> I am using PyArray_FromDimsAndData. If data is passed into this function,
> then the flag OWN_DATA is not set. In array_dealloc, if OWN_DATA is not
> set, then the data pointer of the array object is not freed.
Seeme easy enough...
> I decrement the reference to the array object after the python function
> call. I do not increment it before the function call. The array
Yes, of course. It starts out at one, as for any newly created object.
> I have not done this, but I should. Checking the reference count and
> raising an exception should work in most cases.
It surely beats a crash due to an access to unallocated memory!
Thanks for your comments, it seems clear now what I have to do.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Tue Nov 5 18:45:02 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 5 Nov 96 19:45:02 +0100
Subject: [PYTHON MATRIX-SIG] C API questions
In-Reply-To: <9611051825.AA29044@icf.llnl.gov.llnl.gov> (dubois1@llnl.gov)
Message-ID: <199611051845.TAA03030@ibs.ibs.fr>
> The one part about this discussion I didn't follow was the part about trying
> to make sure Python does not keep a reference to your compiled object. That
> is probably impossible what with traceback stacks and all. Or maybe I didn't
> get it.
But the traceback stack does not keep its references forever, I hope.
Is there maybe a way to force it to give up a certain reference?
> If you do use PyArray_FromDimsAndData you should just follow the normal
> Python rules for reference counting in your extensions and since the count
> started out at 1 it will never get deleted. But if the pointer points to
> something that might not be there later, this is the wrong Python object to
> model it, IMHO.
But what other choice do I have? I want to define a general C-level
interface to some of my code, which might actually be Python code.
So I must construct a C wrapper around the Python functions. But a
C function can't make any assumptions about the existence of its
actual parameters after it returns. It might look like:
double *x = malloc(n*sizeof(double));
do_some_calculation(x, n);
free(x);
with
void do_some_calculation(double *x, int n)
{
PyArrayObject *a = PyArray_FromDimsAndData(.., x, n, ..);
/* call Python function working on a */
}
In that case I must make sure that no access to the memory pointed
to by x can occur after my C wrapper exits.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hugunin@mit.edu Tue Nov 5 19:01:22 1996
From: hugunin@mit.edu (Jim Hugunin)
Date: Tue, 5 Nov 1996 14:01:22 -0500
Subject: [PYTHON MATRIX-SIG] C API questions
Message-ID: <9611051901.AA10047@goldilocks.LCS.MIT.EDU>
> But what other choice do I have? I want to define a general C-level
> interface to some of my code, which might actually be Python code.
> So I must construct a C wrapper around the Python functions. But a
> C function can't make any assumptions about the existence of its
> actual parameters after it returns. It might look like:
>
> double *x = malloc(n*sizeof(double));
> do_some_calculation(x, n);
> free(x);
>
> with
>
> void do_some_calculation(double *x, int n)
> {
> PyArrayObject *a = PyArray_FromDimsAndData(.., x, n, ..);
> /* call Python function working on a */
> }
>
> In that case I must make sure that no access to the memory pointed
> to by x can occur after my C wrapper exits.
My usual solution in this case is to copy data frequently. I assume that
you want to modify the array x in-place here? My own approach in these
situation has been to change my C API to something like:
do_some_calculation(x_in, x_out, n)
where x_out is allocated memory to hold the result.
then in my python code I do:
a = PyArray_FromDims
memcpy x_in into a
do calculation
get back PyArrayObject b
memcpy from b's data area into x_out
For all of the functions I've used, the impact of these memcpy's on my
performance has been too small to measure.
I actually have come to like this approach even for straight C code as it
makes many memory management issues a little bit more straightforward.
-Jim
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Wed Nov 6 09:38:44 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 6 Nov 96 10:38:44 +0100
Subject: [PYTHON MATRIX-SIG] C API questions
In-Reply-To: <9611051901.AA10047@goldilocks.LCS.MIT.EDU> (hugunin@mit.edu)
Message-ID: <199611060938.KAA05501@ibs.ibs.fr>
> My usual solution in this case is to copy data frequently. I assume that
> you want to modify the array x in-place here? My own approach in these
> situation has been to change my C API to something like:
Yes, the data does get modified, but even without modification I wouldn't
think of copying, because the arrays can be very large (meaning a few MB,
and there are several of them around).
But I just got another idea: if before returning I set the array's
shape to something corresponding to a zero length array, then
there should be no way to access its memory area, right? Ignoring
of course buggy C extensions, which I don't care about. To be
even more safe, I could change the data pointer to point to some
small statically allocated area. Or am I missing something?
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hugunin@mit.edu Wed Nov 6 16:17:09 1996
From: hugunin@mit.edu (Jim Hugunin)
Date: Wed, 6 Nov 1996 11:17:09 -0500
Subject: [PYTHON MATRIX-SIG] C API questions
Message-ID: <9611061616.AA24982@goldilocks.LCS.MIT.EDU>
> > My usual solution in this case is to copy data frequently. I assume
that
> > you want to modify the array x in-place here? My own approach in these
> > situation has been to change my C API to something like:
>
> Yes, the data does get modified, but even without modification I wouldn't
> think of copying, because the arrays can be very large (meaning a few MB,
> and there are several of them around).
Okay, I'm still not convinced, see my comments below...
> But I just got another idea: if before returning I set the array's
> shape to something corresponding to a zero length array, then
> there should be no way to access its memory area, right? Ignoring
> of course buggy C extensions, which I don't care about. To be
> even more safe, I could change the data pointer to point to some
> small statically allocated area. Or am I missing something?
Unfortunately, if the user has taken a slice of the array (or indexed into
it if it's more than 1d) they can have an array with a pointer to the data
in your original array and there's no way for you to know this (except of
course for the original array's reference count).
This does appear to be a bit of a problem. I never imagined people would
want to call a NumPy function from generic, non-python-aware C. Could you
explain a little bit more the reasoning behind this design? At first
glance it looks really backwards. I'm also a bit confused as to what you
are doing to these arrays in python code. Most operations on arrays
involve a factor of 2 or so wasted memory (c = a+b). If you're that
concerned about making copies of arrays it's hard for me to imagine what
useful things you could do with them in python code (without writing c-like
code of the form add(a,b,c) which is not meant for frequent use).
-Jim
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Wed Nov 6 16:40:15 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 6 Nov 96 17:40:15 +0100
Subject: [PYTHON MATRIX-SIG] C API questions
In-Reply-To: <9611061616.AA24982@goldilocks.LCS.MIT.EDU> (hugunin@mit.edu)
Message-ID: <199611061640.RAA09510@ibs.ibs.fr>
> Unfortunately, if the user has taken a slice of the array (or indexed into
> it if it's more than 1d) they can have an array with a pointer to the data
> in your original array and there's no way for you to know this (except of
> course for the original array's reference count).
It doesn't help me much to know it, I have to be able to prevent it...
> This does appear to be a bit of a problem. I never imagined people would
> want to call a NumPy function from generic, non-python-aware C. Could you
> explain a little bit more the reasoning behind this design? At first
The application is force field evaluation for molecular systems. There
are a set of "force fields", which are essentially subroutines that
take a list of points as input and return a single value. So far no
problem, but optionally they also add something to selected elements
of one or two arrays passed to them (energy gradients and force
constant matrices).
There will eventually be many of these subroutines, who may even call
one another. And they might be written in Python, C, or Fortran.
So what I need is a unified interface (or rather two, one in Python
and one in C) that makes it possible to call any of these subroutines
without knowing more than its address.
The easiest solution would be to pass around array objects; the C
functions could easily extract the data from them. But I also want the
C interface to be usable from Fortran, and there's the problem.
The only data structure known to Fortran is an array, so I *have*
to pass C-style arrays in my generic interface. And that causes
all the trouble for a force field implemented in Python.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Wed Nov 6 18:53:10 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Wed, 06 Nov 1996 13:53:10 -0500
Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Alpha release of new tutorial for 1.0a5
In-Reply-To: Your message of "Wed, 23 Oct 1996 13:21:12 EDT."
<199610231721.NAA22272@maigret>
References: <199610231721.NAA22272@maigret>
Message-ID: <199611061853.NAA02216@monty>
> For most of the python modules, I think TIM would be good enough.
> For the scientific extensions which are bound to arrive in droves =),
> it'd be nice to have one which allowed math like LaTeX, but also
> allowed it to be displayed in HTML form.
One short-term solution would be to use conditional inclusion. I
believe texinfo supports this -- you can have one version of a
paragraph/sentence/word for one back-end (e.g. TeX), and another for
the others. I *think*...
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Thu Nov 7 02:46:37 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Wed, 06 Nov 1996 21:46:37 -0500
Subject: [PYTHON MATRIX-SIG] Calling all extensions and Sparselib
In-Reply-To: Your message of "Tue, 05 Nov 1996 00:59:55 +0100."
<327E837B.5FFC6C57@linux-magazin.de>
References: <199611042003.OAA19994@cse.unl.edu>
<327E837B.5FFC6C57@linux-magazin.de>
Message-ID: <199611070246.VAA04199@monty>
> Hmm, the dl interface is implemented on all architectures I know.
> When you have dynamic loading of modules running,
> then the dlmodule should work too..
Wish that were true :-(
Unfortunately, dlmodule only knows how to do it on SunOS/Solaris/SGI and
any system that has a compatible implementation. This should probably
be rectified -- the code in importdl.c knows how to do dynamic loading
on many platforms, and knows very little about Python modules...
But for now, you need a sun compatible <dlfcn.h> for dlmodule.
Sorry,
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Thu Nov 7 16:15:42 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 7 Nov 96 17:15:42 +0100
Subject: [PYTHON MATRIX-SIG] Memory leak?
Message-ID: <199611071615.RAA15486@ibs.ibs.fr>
I am beginning to wonder whether there could be a memory leak in some
array functions. I am running a rather simple iterative calculation
which during each iteration creates a few large matrices (always of
exactly the same size) and destroys them in the end (i.e. the
reference count ought to become zero). The results of each iteration
are accumulated in a list, but they are very small. So the whole
program should use a constant amount of memory - but in reality it
starts out at 10 MB during the first iteration and then steadily eats
up more until it crashes the machine (you can imagine how popular I am
among my colleagues after trying this a few times...), which has 128
MB of virtual memory. The highest memory use I have observed is 100
MB, at which point I killed the job.
This may of course be a severe case of memory fragmentation, but even
then I wouldn't be very happy about it. Any ideas how I could analyze
what's wrong?
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hugunin@mit.edu Thu Nov 7 16:40:54 1996
From: hugunin@mit.edu (Jim Hugunin)
Date: Thu, 7 Nov 1996 11:40:54 -0500
Subject: [PYTHON MATRIX-SIG] Memory leak?
Message-ID: <9611071640.AA11796@goldilocks.LCS.MIT.EDU>
> From: Konrad Hinsen <hinsen@ibs.ibs.fr>
> I am beginning to wonder whether there could be a memory leak in some
> array functions.
I'm fairly convinced that there are some memory leaks in the array
functions. I haven't had the time recently to track them down as I'm not
currently being bitten by the bug. The very best way to find these bugs is
to use the program called Purify. I used this myself on earlier versions
of NumPy, but it's gotten much more difficult for me to do this since I
moved my primary development work to NT.
If you don't have access to Purify, I'd suggest the usual debugging
approach of shrinking the problem code to the smallest size you can that
still has the bug. You should be able to identify the specific array
functions that are casuing problems. At this point I'd be happy to look at
the code.
-Jim
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From janssen@parc.xerox.com Thu Nov 7 02:34:40 1996
From: janssen@parc.xerox.com (Bill Janssen)
Date: Wed, 6 Nov 1996 18:34:40 PST
Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Where to go (was Numerical Recipes)
In-Reply-To: <199611070219.VAA16356@dolphin.automatrix.com>
References: <199610211441.JAA00648@darwin.rsoc.rockwell.com>
<199611070219.VAA16356@dolphin.automatrix.com>
Message-ID: <ImUIf04B0KGWI30Vkp@holmes.parc.xerox.com>
Excerpts from ext.python: 6-Nov-96 [PYTHON DOC-SIG] Re: [PYTHO.. Skip
Montanaro@automatri (826)
> (In case it's not clear, I still think migrating to FM is a mistake for
> documentation of free
> software.)
Yes, I tend to agree.
Bill
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Wed Nov 6 18:53:10 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Wed, 06 Nov 1996 13:53:10 -0500
Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Alpha release of new tutorial for 1.0a5
In-Reply-To: Your message of "Wed, 23 Oct 1996 13:21:12 EDT."
<199610231721.NAA22272@maigret>
References: <199610231721.NAA22272@maigret>
Message-ID: <199611061853.NAA02216@monty>
> For most of the python modules, I think TIM would be good enough.
> For the scientific extensions which are bound to arrive in droves =),
> it'd be nice to have one which allowed math like LaTeX, but also
> allowed it to be displayed in HTML form.
One short-term solution would be to use conditional inclusion. I
believe texinfo supports this -- you can have one version of a
paragraph/sentence/word for one back-end (e.g. TeX), and another for
the others. I *think*...
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From skip@calendar.com (Skip Montanaro) Thu Nov 7 02:19:44 1996
From: skip@calendar.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 6 Nov 1996 21:19:44 -0500 (EST)
Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Where to go (was Numerical Recipes)
In-Reply-To: <199610211441.JAA00648@darwin.rsoc.rockwell.com>
References: <199610211441.JAA00648@darwin.rsoc.rockwell.com>
Message-ID: <199611070219.VAA16356@dolphin.automatrix.com>
Robin Friedrich writes:
It should be made clear that the Language Reference and Tutorial
are the only docs converted to FrameMaker.
It should also be made clear that restricting those docs to FrameMaker makes
it doubly difficult to get any help updating them... (In case it's not
clear, I still think migrating to FM is a mistake for documentation of free
software.)
Skip Montanaro | Musi-Cal: http://concerts.calendar.com/
skip@calendar.com | Conference Calendar: http://conferences.calendar.com/
(518)372-5583 | Python: http://www.python.org/
Annoyed with coverage of WWW? http://www.automatrix.com/~skip/diatribe.html
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From janssen@parc.xerox.com Wed Nov 6 20:33:12 1996
From: janssen@parc.xerox.com (Bill Janssen)
Date: Wed, 6 Nov 1996 12:33:12 PST
Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Alpha release of new tutorial for 1.0a5
In-Reply-To: <199611061853.NAA02216@monty>
References: <199610231721.NAA22272@maigret>
<199611061853.NAA02216@monty>
Message-ID: <cmUDM80B0KGW030X5_@holmes.parc.xerox.com>
Excerpts from direct: 6-Nov-96 Re: [PYTHON DOC-SIG] Re: [P.. Guido van
Rossum@cnri.re (539)
> One short-term solution would be to use conditional inclusion. I
> believe texinfo supports this -- you can have one version of a
> paragraph/sentence/word for one back-end (e.g. TeX), and another for
> the others. I *think*...
Yes; here's the relevant section from the TeXinfo manual:
Using Ordinary TeX Commands
===========================
Inside a region delineated by `@iftex' and `@end iftex', you can
embed some PlainTeX commands. Info will ignore these commands since
they are only in that part of the file which is seen by TeX. You can
write the TeX commands as you would write them in a normal TeX file,
except that you must replace the `\' used by TeX with an `@'. For
example, in the `@titlepage' section of a Texinfo file, you can use the
TeX command `@vskip' to format the copyright page. (The `@titlepage'
command causes Info to ignore the region automatically, as it does with
the `@iftex' command.)
However, many features of PlainTeX will not work, as they are
overridden by features of Texinfo.
You can enter PlainTeX completely, and use `\' in the TeX commands,
by delineating a region with the `@tex' and `@end tex' commands. (The
`@tex' command also causes Info to ignore the region, like the `@iftex'
command.)
For example, here is a mathematical expression written in PlainTeX:
@tex
$$ \chi^2 = \sum_{i=1}^N
\left (y_i - (a + b x_i)
\over \sigma_i\right)^2 $$
@end tex
The output of this example will appear only in a printed manual. If
you are reading this in Info, you will not see anything after this
paragraph.
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From busby@icf.llnl.gov Fri Nov 8 22:43:24 1996
From: busby@icf.llnl.gov (L. Busby)
Date: Fri, 8 Nov 96 14:43:24 PST
Subject: [PYTHON MATRIX-SIG] ANNOUNCE: June wishlist patches, new Gist released
Message-ID: <9611082243.AA17570@icf.llnl.gov.llnl.gov>
I've just posted an announcement to comp.lang.python regarding a set of
three large patches to Python-1.4, and a new release of Gist. Numeric
users may take special interest. If you do not or cannot see the
posting there, please let me know. You can pick up the new stuff at
ftp://icf.llnl.gov/pub/python/busby/patches.961108.tgz
ftp://icf.llnl.gov/pub/python/busby/pygist-1.2.tgz
Lee
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From viennet@ura1507.univ-paris13.fr Sun Nov 10 11:16:55 1996
From: viennet@ura1507.univ-paris13.fr (Emmanuel Viennet)
Date: Sun, 10 Nov 1996 11:16:55 GMT
Subject: [PYTHON MATRIX-SIG] .flat attribute
Message-ID: <199611101116.LAA16555@univ-paris13.fr>
It seems that the attribute .flat of an array is usable only if the
array is contiguous.
(ValueError: flattened indexing only available for contiguous array)
This point should be mentionned in the tutorial.
Moreover, since a function can't know if the array it will process will be
or not contiguous, they are very few cases where .flat is usefull.
Or did I miss something ?
Emmanuel
--
Emmanuel Viennet: <viennet@ura1507.univ-paris13.fr>
LIPN - Institut Galilee - Universite Paris-Nord
93430 Villetaneuse - France
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hugunin@mit.edu Mon Nov 11 20:00:11 1996
From: hugunin@mit.edu (Jim Hugunin)
Date: Mon, 11 Nov 1996 15:00:11 -0500
Subject: [PYTHON MATRIX-SIG] .flat attribute
Message-ID: <9611112124.AA09565@goldilocks.LCS.MIT.EDU>
> It seems that the attribute .flat of an array is usable only if the
> array is contiguous.
> (ValueError: flattened indexing only available for contiguous array)
>
> This point should be mentionned in the tutorial.
True. The preferred way to get a flattened array is the ravel function.
This should always work.
> Moreover, since a function can't know if the array it will process will
be
> or not contiguous, they are very few cases where .flat is usefull.
> Or did I miss something ?
I start many of my functions with the code:
a = array(a, Float)
This quarantees that I have a contiguous array of C doubles and that I can
modify this array without affecting the arrays held by the caller of the
function. The obvious problem with this is that it adds an array copy to
every such function. Where this is a problem I don't do it.
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From cgw@pgt.com Wed Nov 13 22:03:33 1996
From: cgw@pgt.com (Charles G Waldman)
Date: Wed, 13 Nov 1996 17:03:33 -0500
Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction"
Message-ID: <9611132203.AA12113@reactor.pgt>
Can anyone explain to me whether the following is a feature or a bug?
Python 1.4 (Oct 31 1996) [GCC 2.7.2]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import ni,Numeric
>>> def g(a):
... return a
...
>>> x = Numeric.fromfunction(g,[10])
>>> x
0 1 2 3 4 5 6 7 8 9
>>> len(x)
10
>>>
>>>
>>> def g(a):
... return 5
...
>>> x = Numeric.fromfunction(g,[10])
>>> x
5
>>> len(x)
Traceback (innermost last):
File "<stdin>", line 1, in ?
TypeError: len() of unsized object
>>>
>>>
>>> def g(a):
... return a*0 + 5
...
>>> x = Numeric.fromfunction(g,[10])
>>> x
5 5 5 5 5 5 5 5 5 5
>>> len(x)
10
>>>
This has me quite puzzled. If the initializer function "g" returns a constant,
I get a scalar rather than an array, even though I have asked for a shape
of [10]. If the function "g" uses its argument (here in a trivial way,
multiplied by zero!) then I get the behavior I expect.
I first noticed this trying to use "fromfunction" with a random-number
generator, to create arrays of random numbers. My code currently works,
but it has the strange "multiply by zero" stuck in there, so it looks
pretty weird. I'd like to have a better understanding of what's going
on. Thanks in advance for any comments!
=============================
| Charles G. Waldman |
| Princeton Gamma-Tech, Inc.|
| cgw@pgt.com |
=============================
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Thu Nov 14 09:23:04 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 14 Nov 96 10:23:04 +0100
Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction"
In-Reply-To: <9611132203.AA12113@reactor.pgt> (message from Charles G Waldman
on Wed, 13 Nov 1996 17:03:33 -0500)
Message-ID: <199611140922.KAA11272@ibs.ibs.fr>
> Can anyone explain to me whether the following is a feature or a bug?
>
> Python 1.4 (Oct 31 1996) [GCC 2.7.2]
> Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
> >>> import ni,Numeric
> >>> def g(a):
> ... return a
> ...
> >>> x = Numeric.fromfunction(g,[10])
> >>> x
> 0 1 2 3 4 5 6 7 8 9
> >>> len(x)
> 10
> >>>
> >>>
> >>> def g(a):
> ... return 5
> ...
> >>> x = Numeric.fromfunction(g,[10])
> >>> x
> 5
Feature, though not a well-known one. fromfunction() works correctly
only if the function
1) uses all its arguments
2) consists only of scalar operations defined on arrays (i.e. elementary
arithmetic operations and functions from umath).
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From da@maigret.cog.brown.edu Thu Nov 14 15:06:52 1996
From: da@maigret.cog.brown.edu (David Ascher)
Date: Thu, 14 Nov 1996 10:06:52 -0500 (EST)
Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction"
In-Reply-To: <199611140922.KAA11272@ibs.ibs.fr> from "Konrad Hinsen" at Nov 14, 96 10:23:04 am
Message-ID: <199611141506.KAA09924@maigret>
> Feature, though not a well-known one. fromfunction() works correctly
> only if the function
> 1) uses all its arguments
> 2) consists only of scalar operations defined on arrays (i.e. elementary
> arithmetic operations and functions from umath).
This is in the tutorial:
---
Currently, the function which is called by fromfunction can only use
numerical operations and ufuncs on its arguments. It cannot, for
example, use if ... else: ... constructs. This is due to the fact that
fromfunction does operations "en masse" to speed up evaluation. If you
need to fill an array with the result of a function which does not meet
these criteria, you can always use a function like:
---
I will also add in the next release of the tutorial code from Konrad
which removes these limitations (at the expense of speed, alas).
--david
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Thu Nov 14 15:42:08 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 14 Nov 96 16:42:08 +0100
Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction"
In-Reply-To: <199611141506.KAA09924@maigret> (da@maigret.cog.brown.edu)
Message-ID: <199611141541.QAA13989@ibs.ibs.fr>
> I will also add in the next release of the tutorial code from Konrad
> which removes these limitations (at the expense of speed, alas).
To the extent of making a general-purpose function worse than a simple
nested loop that assigns the elements. So maybe that may be the
best solution in many cases.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28
Institut de Biologie Structurale | Fax: +33-76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From kschalm@geog.ubc.ca Fri Nov 15 21:12:01 1996
From: kschalm@geog.ubc.ca (Kyle Schalm)
Date: Fri, 15 Nov 1996 13:12:01 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] ncmodule memory fix
Message-ID: <199611152112.NAA04134@peacock.geog.ubc.ca>
the netcdf module (array version) has a memory fix. you can apply the
patch below or download the new version of ncmodule.c from
http://www.geog.ubc.ca/~kschalm/
here's the patch (< = new version, > = old version):
1521c1521
< PyObject *list, *m, *t, *ret, *a; /* array taken from arg */
---
> PyObject *list, *m, *t, *a; /* array taken from arg */
1591,1593c1591
< ret = PyArray_FromDimsAndData(self->ndims, dimlen, array_type, t_data);
< if (ret != NULL) ((PyArrayObject *) ret)->flags |= OWN_DATA;
< return ret;
---
> return PyArray_FromDimsAndData(self->ndims, dimlen, array_type, t_data);
--Kyle
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From jhauser@ifm.uni-kiel.de Sat Nov 16 14:16:31 1996
From: jhauser@ifm.uni-kiel.de (janko hauser)
Date: Sat, 16 Nov 1996 15:16:31 +0100
Subject: [PYTHON MATRIX-SIG] When 1.06{a,b}?
Message-ID: <9611161416.AA03998@caesar.ifm.uni-kiel.de>
Hello I want to build a new python with new gistmodule to get
commandline-history in the interpreter. Is the new version comming in
the near future? If not what are the necessary changes regarding
``ellipsis''? This is the main problem at the moment to build Numpy
with Python1.4, isn't it?
TIA
__Janko
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From viennet@ura1507.univ-paris13.fr Mon Nov 18 12:00:38 1996
From: viennet@ura1507.univ-paris13.fr (Emmanuel Viennet)
Date: Mon, 18 Nov 1996 12:00:38 GMT
Subject: [PYTHON MATRIX-SIG] Saving HUGE arrays
Message-ID: <199611181200.MAA11032@univ-paris13.fr>
I am working with huge arrays (size comparable to the total swap
space of the machine, ie ~ 100Mo).
Currently, we write theses arrays to disk using the following
procedure:
fp.write( some_header_info )
fp.write( arr.tostring() )
or in some case:
fp.write( arr.byteswapped().tostring() )
This works but duplicates the array data to get a string.
I know Python strings are immutables, so sharing data between a string
and an array is prohibited.
Should I write a C simple function
write_array( arr, fp, byte_order )
directly writing the data to a file, swapping bytes if requested,
or is there a more standard solution ?
Thanks
Emmanuel
--
Emmanuel Viennet: <viennet@ura1507.univ-paris13.fr>
LIPN - Institut Galilee - Universite Paris-Nord
93430 Villetaneuse - France
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Thu Nov 21 06:32:15 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Thu, 21 Nov 1996 01:32:15 -0500
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
References: <199611181200.MAA11032@univ-paris13.fr>
Message-ID: <199611210632.BAA08397@monty>
(Crossposting from the matrix-sig to the Pythmain list.)
> I am working with huge arrays (size comparable to the total swap
> space of the machine, ie ~ 100Mo).
> Currently, we write theses arrays to disk using the following
> procedure:
> fp.write( some_header_info )
> fp.write( arr.tostring() )
> or in some case:
> fp.write( arr.byteswapped().tostring() )
>
> This works but duplicates the array data to get a string.
>
> I know Python strings are immutables, so sharing data between a string
> and an array is prohibited.
> Should I write a C simple function
>
> write_array( arr, fp, byte_order )
>
> directly writing the data to a file, swapping bytes if requested,
>
> or is there a more standard solution ?
Hmm, interesting. Jack Jansen just mailed me an idea for a standard
way to access "buffer-like" Python objects as linear sequences of
bytes in Python, especially for I/O purposes. This should work for
strings, arrays from the old array module, and numerical arrays, and
other extension types that have an internal representation that is a
contiguous sequence of bytes. (Jack's motivation was that it would
make extensions for manipulating sound data easier to write; it seems
the same problem as stated here.)
Does the matrix community think this would be useful? Any thoughts on
the form of the interface?
I would presume the interface could be something like
int PyBuffer_GetInfo(object *o, const char **address, int *length);
to get the address and size of a "buffer" object into a C char pointer
(or should that be void???) and length. The return value would be -1
if the object was not "bufferable". I guess a PyBuffer_Check(o) call
should also be available. For *mutable* buffer objects, a version
that returns a non-const pointer could be provided (but for strings
this would fail) so one could read data *into* a buffer object.
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From johann@physics.berkeley.edu Thu Nov 21 07:59:04 1996
From: johann@physics.berkeley.edu (Johann Hibschman)
Date: Wed, 20 Nov 1996 23:59:04 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] gist setup help?
Message-ID: <Pine.SUN.3.91.961120235112.28251A-100000@physics20.Berkeley.EDU>
Hi all,
I'm trying to get the gist module to work on my laptop running Linux, but
it keeps dumping core on me. I can compile it and run the demo, but it
crashes after displaying the first window with some sort of an X error.
I have yorick working fine, and the rest of the Numeric module seems to
work, but something's going terribly wrong.
Any ideas? The only thing I can think of was that I changed the X
directory path from /usr/X11R5 (or some such) to /usr/X11R6/lib in the
lines for Setup from Setup.forgist.
I'm running Python 1.4 with NumPy 1.0a5 (with Ellipses->Ellipsis), with
the three patches from the gist distribution, and gistmodule 1.2.
- Johann
Johann Hibschman
johann@physics.berkeley.edu
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From Emmanuel.Viennet@laforia.ibp.fr Thu Nov 21 11:00:17 1996
From: Emmanuel.Viennet@laforia.ibp.fr (VIENNET Emmanuel 48.06.27.97 Equipe Gallinari)
Date: Thu, 21 Nov 1996 12:00:17 +0100
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: <199611210632.BAA08397@monty>
References: <199611181200.MAA11032@univ-paris13.fr>
<199611210632.BAA08397@monty>
Message-ID: <199611211100.MAA04308@lpia2.ibp.fr>
> I would presume the interface could be something like
>
> int PyBuffer_GetInfo(object *o, const char **address, int *length);
>
> to get the address and size of a "buffer" object into a C char pointer
> (or should that be void???) and length. The return value would be -1
> if the object was not "bufferable". I guess a PyBuffer_Check(o) call
> should also be available. For *mutable* buffer objects, a version
> that returns a non-const pointer could be provided (but for strings
> this would fail) so one could read data *into* a buffer object.
Cool. This would allow very efficient implementation of matrix I/O
packages.
I presume PyBuffer_GetInfo() should fail on non-contiguous arrays ?
PyBuffer_GetInfo() must check if the buffer is available read-only
(immutable objects) or read/write (mutable, e.g. contiguous arrays).
We could add it a mode argument, ala fopen() ('r', 'w').
Emmanuel
--
Emmanuel Viennet: <viennet@ura1507.univ-paris13.fr>
LIPN - Institut Galilee - Universite Paris-Nord
93430 Villetaneuse - France
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Thu Nov 21 12:04:04 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 21 Nov 96 13:04:04 +0100
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: <199611211100.MAA04308@lpia2.ibp.fr>
(Emmanuel.Viennet@laforia.ibp.fr)
Message-ID: <199611211203.NAA14724@ibs.ibs.fr>
> I presume PyBuffer_GetInfo() should fail on non-contiguous arrays ?
That would make it useless for many applications, since non-contiguous
arrays can occur anywhere.
> PyBuffer_GetInfo() must check if the buffer is available read-only
> (immutable objects) or read/write (mutable, e.g. contiguous arrays).
>
> We could add it a mode argument, ala fopen() ('r', 'w').
How about a stream-like protocol, i.e. implementing read() and write()?
That could be done for non-contiguous arrays, and still be efficient.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From Zachary_Roadhouse@brown.edu Thu Nov 21 13:48:18 1996
From: Zachary_Roadhouse@brown.edu (Zachary_Roadhouse@brown.edu)
Date: Thu, 21 Nov 1996 08:48:18 -0500 (EST)
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: <199611211100.MAA04308@lpia2.ibp.fr>
Message-ID: <XFMail.961121085256.Zachary_Roadhouse@brown.edu>
On 21-Nov-96 Emmanuel.Viennet@laforia.ibp.fr wrote:
>Cool. This would allow very efficient implementation of matrix I/O
>packages.
>
>I presume PyBuffer_GetInfo() should fail on non-contiguous arrays ?
I agree with Konrad here. What would be needed for a non-continguous
array would just be a little more book keeping. Read and write
methods would be great! This feature would make it easier to pass
data around to my labs non-Python crew. It would also might make it
easier to transform a Numeric array into a PIL image (a common task
in my group) by giving the Image a "file" to work with -- of course
it'll be faster... Thumbs up here.
- Zack
E-MAIL: Zachary_Roadhouse@brown.edu WEB: http://althor.netspace.org/~zack/
Brown University, Box 220, Providence, RI 02912
Phone: (401) 863 - 5435
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Thu Nov 21 16:26:45 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Thu, 21 Nov 1996 11:26:45 -0500
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: Your message of "Thu, 21 Nov 1996 13:04:04 +0100."
<199611211203.NAA14724@ibs.ibs.fr>
References: <199611211203.NAA14724@ibs.ibs.fr>
Message-ID: <199611211626.LAA09612@monty>
> How about a stream-like protocol, i.e. implementing read() and write()?
> That could be done for non-contiguous arrays, and still be efficient.
This makes some sense, except that it wouldn't (really) help for the
situation where you have a HUGE array (HUGE == too big to copy in
virtual memory) and you need to write it to disk. If the array is
contiguous, a single write() system call could write the entire thing
to disk.
Next try (though this is getting complicated :-( ): how about a
"sequence-of-buffers" interface. E.g.:
long PyBuffer_Get(object *o, int i, const void **addres);
would return the length of the i'th contiguous chunk, and store its
address via *address; when there are no more chunks, it can return 0,
and for non-buffering objects, it can return -1 and set an exception.
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Thu Nov 21 17:22:54 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 21 Nov 96 18:22:54 +0100
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: <199611211626.LAA09612@monty> (message from Guido van Rossum on
Thu, 21 Nov 1996 11:26:45 -0500)
Message-ID: <199611211722.SAA17160@ibs.ibs.fr>
> > How about a stream-like protocol, i.e. implementing read() and write()?
> > That could be done for non-contiguous arrays, and still be efficient.
>
> This makes some sense, except that it wouldn't (really) help for the
> situation where you have a HUGE array (HUGE == too big to copy in
> virtual memory) and you need to write it to disk. If the array is
> contiguous, a single write() system call could write the entire thing
> to disk.
One solution would be to add some nice stream protocol to Python.
Files would be one kind of stream, memory blocks another one.
Then file.write(memory_block) should do the following:
1) The file object asks the memory block object for the next
chunk of data (however large), imposing a maximum (which
would actually be infinite in case of files).
2) The memory block object provides a pointer and length indicator for
such a block and remembers where to continue, or an end-of-stream
indication when there is no more data.
3) The file object writes the data to the disk file.
4) Goto 1.
Such a protocol would transfer data in the largest blocks that both
objects can handle.
> Next try (though this is getting complicated :-( ): how about a
> "sequence-of-buffers" interface. E.g.:
I am not too fond of solutions that just try to work around one
special problem. But of course it's better than raising exceptions
for arrays.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Thu Nov 21 19:13:53 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Thu, 21 Nov 1996 14:13:53 -0500
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: Your message of "Thu, 21 Nov 1996 18:22:54 +0100."
<199611211722.SAA17160@ibs.ibs.fr>
References: <199611211722.SAA17160@ibs.ibs.fr>
Message-ID: <199611211913.OAA09984@monty>
> One solution would be to add some nice stream protocol to Python.
> Files would be one kind of stream, memory blocks another one.
> Then file.write(memory_block) should do the following:
>
> 1) The file object asks the memory block object for the next
> chunk of data (however large), imposing a maximum (which
> would actually be infinite in case of files).
>
> 2) The memory block object provides a pointer and length indicator for
> such a block and remembers where to continue, or an end-of-stream
> indication when there is no more data.
>
> 3) The file object writes the data to the disk file.
>
> 4) Goto 1.
>
> Such a protocol would transfer data in the largest blocks that both
> objects can handle.
>
> > Next try (though this is getting complicated :-( ): how about a
> > "sequence-of-buffers" interface. E.g.:
>
> I am not too fond of solutions that just try to work around one
> special problem. But of course it's better than raising exceptions
> for arrays.
My proposal exactly matches your item (2), so we may be on to
something...
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Fri Nov 22 10:01:15 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Fri, 22 Nov 96 11:01:15 +0100
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: <199611211913.OAA09984@monty> (message from Guido van Rossum on
Thu, 21 Nov 1996 14:13:53 -0500)
Message-ID: <199611221000.LAA19591@ibs.ibs.fr>
> > > Next try (though this is getting complicated :-( ): how about a
> > > "sequence-of-buffers" interface. E.g.:
> >
> > I am not too fond of solutions that just try to work around one
> > special problem. But of course it's better than raising exceptions
> > for arrays.
>
> My proposal exactly matches your item (2), so we may be on to
> something...
But the other items are also important. Suppose you want to
transfer data between two streams, which each have restrictions
on buffer size. Something must mediate between the two.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From guido@CNRI.Reston.Va.US Fri Nov 22 16:07:43 1996
From: guido@CNRI.Reston.Va.US (Guido van Rossum)
Date: Fri, 22 Nov 1996 11:07:43 -0500
Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays
In-Reply-To: Your message of "Fri, 22 Nov 1996 11:01:15 +0100."
<199611221000.LAA19591@ibs.ibs.fr>
References: <199611221000.LAA19591@ibs.ibs.fr>
Message-ID: <199611221607.LAA11933@monty>
> > My proposal exactly matches your item (2), so we may be on to
> > something...
>
> But the other items are also important. Suppose you want to
> transfer data between two streams, which each have restrictions
> on buffer size. Something must mediate between the two.
Hm. A proper stream protocol should never refuse to write more data
than what fits in its buffer (it should just flush the buffer and
refill it until it has processed all the data). When asked to read
the next chunk it may return as much or as little as it has available
in its buffer.
But anyway, a stream protocol needs serious thought before it is
added, because it has big ramifications. A buffer interface could be
added with very little effort and could be immediately useful.
--Guido van Rossum (home page: http://www.python.org/~guido/)
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From johann@physics.berkeley.edu Mon Nov 25 19:47:57 1996
From: johann@physics.berkeley.edu (Johann Hibschman)
Date: Mon, 25 Nov 1996 11:47:57 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] Guidance for extensions?
Message-ID: <Pine.SUN.3.91.961125114557.10432B-100000@physics20.Berkeley.EDU>
[ I sent this to the comp.lang.python as well, but I think you people
are more likely to be able to answer my questions... ]
Hello all,
I've recently decided that Numeric Python all by itself is too slow
for a project that I'm working on, so I'm trying to learn how to
extend python. However, the docs I've found are rather confusing.
I want to take a C++ program with classes like
class Solver {
public:
int numPoints;
Vector density, temperature;
void DoTimeSteps( int numSteps );
};
and arrange a python module which would mirror the structure of the
class. I want to be able to use python to set the variables, then
call pass the arrays from python to the C++ object, call the C++
object to advance the solution by some number of timesteps, then pass
the arrays back from the C++ object to the python object, so I can
interactively play with them.
This seems like it should be easy to do, but I'm not sure where to
begin. The docs cover modules with simple functions fairly well, but
I'm not sure how to construct C++ classes so they work, and I don't
know how to transfer Numeric python arrays (or even normal python
lists) to and from my program.
Any suggestions, pointers, comments?
- Johann
Johann Hibschman
johann@physics.berkeley.edu
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From janne@avocado.pc.helsinki.fi Tue Nov 26 16:25:23 1996
From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen)
Date: 26 Nov 1996 18:25:23 +0200
Subject: [PYTHON MATRIX-SIG] A bug in 1.0a5 (ones, concatenate)
Message-ID: <oaafs4bz7w.fsf@avocado.pc.helsinki.fi>
This produces a segmentation fault:
concatenate([ones([0,10]),ones([1,10])])
This does not:
concatenate([zeros([0,10]),ones([1,10])])
I'm using Linux, Python 1.4 and 1.0a5 with fixed ellipses.
--
Janne
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From hinsen@ibs.ibs.fr Tue Nov 26 18:14:10 1996
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 26 Nov 96 19:14:10 +0100
Subject: [PYTHON MATRIX-SIG] A bug in 1.0a5 (ones, concatenate)
In-Reply-To: <oaafs4bz7w.fsf@avocado.pc.helsinki.fi> (message from Janne
Sinkkonen on 26 Nov 1996 18:25:23 +0200)
Message-ID: <199611261812.TAA09191@ibs.ibs.fr>
> This produces a segmentation fault:
>
> concatenate([ones([0,10]),ones([1,10])])
>
> This does not:
>
> concatenate([zeros([0,10]),ones([1,10])])
>
> I'm using Linux, Python 1.4 and 1.0a5 with fixed ellipses.
Same under HP/UX. And there are other operations with empty arrays
that cause crashes.
In case you wonder what the difference between ones() and zeros()
is: zeros() is implemented in C; ones() is implemented in Python
as, essentially, zero()+1. It might be the addition that ultimately
causes the crash.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From barrett@compass.gsfc.nasa.gov Tue Nov 26 18:31:07 1996
From: barrett@compass.gsfc.nasa.gov (Paul Barrett)
Date: Tue, 26 Nov 1996 13:31:07 -0500
Subject: [PYTHON MATRIX-SIG] NumPy benchmarks
Message-ID: <199611261831.NAA25558@compass.gsfc.nasa.gov>
Greetings,
I've been doing some benchmarks of the fundamental array operations,
e.g. '+', '*', 'cos', 'log', etc., and making comparisons to C code
and IDL (RSI's Interactive Data Language). I thought it would be
useful to have some numbers to show people.
The results on a SPARCstation 20 with 64 MB RAM are in MFIPS (millions
of floating-pointing instructions per second) where the array type is
Float32 and lengths are in millions of elements. Hence, higher numbers
are better.
Addition and multiplication operations:
Length C IDL Python
0.1 6.67 6.67 5.00
0.3 4.45 4.85 3.41
1.0 4.52 4.86 3.26
3.0 4.55 4.87 3.02
5.0 0.72 0.36 0.51
Cosine operation:
Length C IDL Python
0.1 0.51 0.51 0.53
0.3 0.57 0.50 0.52
1.0 0.57 0.50 0.52
3.0 0.20 0.13 0.19
5.0 0.15 0.09 0.15
Tangent operation:
Length C IDL Python
1.0 0.38 0.44 0.35
Logarithm operation:
Length C IDL Python
1.0 0.75 0.70 0.65
1) As the numbers indicate, the binary operations show room for
improvement. They are about 33% slower than the C and IDL code. The
unary operations are typically within 10% and therefore are
satisfactory. Does anyone have any suggestions about how the binary
ops can be improved?
2) Also note that the numbers become worse as the arrays become a
sizeable fraction of the computer's RAM size. The addition of two
arrays in a third array of 5 million elements each requires 60 MB of
out of 64 MB of RAM. Thus the results become dependant on the time
needed to swap memory. At present, it's not clear to me if
improvements can be made in this area, but it would be a nice feather
in Python's cap if improvements could be made for very large arrays.
3) Having had sometime to work with NumPy recently, I have three
comments about the implementation:
The first is the use of 'arccos', 'arcsin', etc. instead of using
'acos', 'asin', etc. This seems to me to be both inconsistent with
Python's math module which uses 'acos', 'asin', etc. and with common
practice. Is there a reason for this?
The second is about the 'arrayrange' method. It is very slow for very
large arrays. During my initial tests, I found Python to be very slow
compared to C and IDL. I then discovered the reason to be my use of
arrayrange to create arrays. As the source code notes, this method
should be converted to C, since it can be used heavily in some
numerical algorithms.
And the third is the difference between umathmodule.c and
fast_umathmodule.c. Currently there is none. Will someone please
outline what they should be?
Cheers,
Paul
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
| Paul Barrett - Astrophysicist - Universities Space Research Association |
| Compton Observatory Science Support Center |
| NASA/Goddard SFC phone: 301-286-1108 "Cracking toast, Gromit" |
| Code 660.1, FAX: 301-286-1681 says Wallace. |
| Greenbelt,MD 20771 barrett@compass.gsfc.nasa.gov (in The Wrong Trousers) |
| http://lheawww.gsfc.nasa.gov/users/barrett/CV.html |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From johann@physics.berkeley.edu Tue Nov 26 19:05:53 1996
From: johann@physics.berkeley.edu (Johann Hibschman)
Date: Tue, 26 Nov 1996 11:05:53 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] SWIG, shadow classes, and NumPy
Message-ID: <Pine.SUN.3.91.961126104839.13912A-100000@physics20.Berkeley.EDU>
Hi all,
In my continuing struggles to get my hydro code working and working
fast, I have lapsed into straight C++ with a Python shadow class generated
by SWIG. So far that works beautifully; I can get at all of my data and
I can run the computational methods of my class.
The one problem is that all of the vectors come out as specialized objects
for which I have to define the __getitem__ and __setitem__ methods
explicitly in the SWIG .i file. It would be nice to be able to use all of
the interface options available to standard NumPy arrays (math operations,
assignment by a scalar, slices, etc.) and have these operations afffect
the actual data in my C++ class.
One way to do this would be to mirror all of the data arrays as NumPy
arrays in the shadow class, but this leads to problems with the shadow
class getting out of synch with the C++ class. I'm not sure exactly how
I'd implement this, but I'm confident that I could after reading through
the NumPy source code a few times. (Not that I really want to do that.)
Is there another, more clever way which someone more familiar with the ins
and outs of the NumPy innards can see to do this? Could I use a NumPy
array in the native C++ code? Could I create a class wrapper for a NumPy
array so it would be usable by C++? Could I delude a NumPy array object
into thinking that the block of data in the C++ vector belonged to it
instead, without having the NumPy array attempt to delete the data at some
point? Could I introduce SWIG directives that would somehow map from
one to the other at runtime?
Thanks,
- Johann
Johann Hibschman
johann@physics.berkeley.edu
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From tbyang@icf.llnl.gov Tue Nov 26 20:37:00 1996
From: tbyang@icf.llnl.gov (Tser-Yuan (Brian) Yang)
Date: Tue, 26 Nov 1996 12:37:00 -0800 (PST)
Subject: [PYTHON MATRIX-SIG] Re: SWIG, shadow classes, and NumPy
Message-ID: <Pine.SUN.3.95.961126123258.17966A-100000@icf.llnl.gov>
Since SWIG shadows a pointer with a Python string object, I think
the following can be useful:
In your C++ extension, implement a method 'Ptr_to_PyArray' that takes three
arguments, i.e., a pointer object for the data, a pointer object for
the shape, and a Python integer object for the number of dimension, and
returns a PyArray object of the Numeric module. This can be done with
the API 'PyArray_FromDimsAndData' in 'arrayobject.c'. An inverse method
'PyArray_to_Ptr' that takes a PyArray object and returns a tuple may
be very useful, too.
I have been thinking about implementing these interface functions in
one of our applications, but I am putting it off because my current
project don't need them.
Brian Yang
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================
From chris.chase@jhuapl.edu Fri Nov 29 20:38:39 1996
From: chris.chase@jhuapl.edu (Chris Chase SRM)
Date: Fri, 29 Nov 1996 15:38:39 -0500
Subject: [PYTHON MATRIX-SIG] changes to fix __array__()
Message-ID: <199611292041.PAA25686@python.org>
I am using NumPy-1.0a5.tar and Python 1.4.
I was trying to use the __array__(self, typecode=None) interface for
getting an object as an array and found that it did not work. I
traced the problem to array_fromobject() in arrayobject.c.
I have made the necessary changes to array_fromobject() in
arrayobject.c. I also made corrections to refecrence handling of
Python objects in the same function (according to Lutz's PP book both
PyEval_CallObject() and PyObject_GetAttrString() pass pass owenership
of result, i.e. it is already INCREF'd and must be DECREF'd when
finished).
As a result of all this I found that asarray() did not use
PyArray_FromObject() but instead copies via PyArray_CopyFromObject
when using the __array__() interface. To make it use
PyArray_FromObject(), I moved asarray() in multiarray.c instead of in
Numeric.py.
An example of the fixed behavior of the __array__ interface and
asarray():
class a:
def __init__(self):
self.a = arange(8)
def __array__(self,t=None):
if t: return asarray(self.a,t)
return asarray(self.a)
b = a()
c = array(b)
c[2] = -9
>>> c
0 1 -9 3 4 5 6 7
>>> b.a
0 1 2 3 4 5 6 7
d = asarray(b)
d[2] = -9
>>> b.a
0 1 -9 3 4 5 6 7
According to the documentation for array the interface is given as
"array(sequence, typecode=None)" which seems to indicate that None is
valid value for typecode. However, None is not a valid value and
typecode is not a keyword argument. The documentation should be
changed. Although, I think that it would be better to let typecode be
a possible keyword argument and let None be a valid value that gets
mapped to PyArray_NOTYPE (in arrayobject.h).
Anyway, for the __array__ and asarray changes I modified
arrayobject.c, multiarray.c and Numeric.py. The context diffs created
in the NumPy directory are below, which I believe are suitable for the
patch program, are below.
Chris Chase
===================================================================
RCS file: arrayobject.c,v
retrieving revision 1.1
diff -c5 -r1.1 arrayobject.c
*** arrayobject.c 1996/11/28 07:10:25 1.1
--- arrayobject.c 1996/11/29 19:13:11
***************
*** 1612,1641 ****
}
#define ByCopy 1
#define Contiguous 2
! PyObject *array_fromobject(PyObject *op, int type, int min_depth, int max_depth, int flags) {
! PyObject *r;
!
r = NULL;
! if (!PyArray_Check(op) && PyObject_HasAttrString(op, "__array__")) {
PyObject *ap, *arglist;
if (type == PyArray_NOTYPE) {
arglist = Py_BuildValue("()");
} else {
arglist = Py_BuildValue("(c)", type);
}
! ap = PyObject_GetAttrString(op, "__array__");
! r = PyEval_CallObject(ap, arglist);
Py_DECREF(arglist);
!
! Py_XINCREF(r);
! }
!
if (type == PyArray_NOTYPE) {
type = PyArray_ObjectType(op, 0);
}
if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT ||
--- 1612,1646 ----
}
#define ByCopy 1
#define Contiguous 2
! PyObject *array_fromobject(PyObject *op_in, int type, int min_depth, int max_depth, int flags) {
! PyObject *r, *op;
!
r = NULL;
! if (!PyArray_Check(op_in) && PyObject_HasAttrString(op_in, "__array__")) {
! /* __array__(self, type=None) method interface
! for getting an object as an array. */
PyObject *ap, *arglist;
if (type == PyArray_NOTYPE) {
arglist = Py_BuildValue("()");
} else {
arglist = Py_BuildValue("(c)", type);
}
! ap = PyObject_GetAttrString(op_in, "__array__");
! op = PyEval_CallObject(ap, arglist);
! Py_DECREF(ap);
Py_DECREF(arglist);
! if (op == NULL) return NULL;
! } else {
! op = op_in;
! Py_INCREF(op);
! }
!
if (type == PyArray_NOTYPE) {
type = PyArray_ObjectType(op, 0);
}
if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT ||
***************
*** 1648,1673 ****
Py_INCREF(op);
r = op;
}
} else {
if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num;
! if ((r = PyArray_Cast((PyArrayObject *)op, type)) == NULL) return NULL;
}
} else {
r = Array_FromSequence(op, type, min_depth,max_depth);
! if (r == NULL) {
! if (min_depth <= 0) {
! PyErr_Clear();
! r = PyArray_FromScalar(op, type);
! } else {
! return NULL;
! }
}
}
if (r == NULL) return NULL;
! if(!PyArray_Check(r)) {
PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array");
return NULL;
}
if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) {
Py_DECREF(r);
--- 1653,1677 ----
Py_INCREF(op);
r = op;
}
} else {
if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num;
! r = PyArray_Cast((PyArrayObject *)op, type);
}
} else {
r = Array_FromSequence(op, type, min_depth,max_depth);
! if (r == NULL && min_depth <= 0) {
! PyErr_Clear();
! r = PyArray_FromScalar(op, type);
}
}
+ /* finished with op */
+ Py_DECREF(op);
+
if (r == NULL) return NULL;
! if (!PyArray_Check(r)) {
PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array");
return NULL;
}
if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) {
Py_DECREF(r);
rcsdiff: RCS/multiarray.c,v: No such file or directory
rcsdiff: Numeric/RCS/Numeric.py,v: No such file or directory
custer [121]> ../misc/makepatch
../misc/makepatch
===================================================================
RCS file: arrayobject.c,v
retrieving revision 1.1
diff -c5 -r1.1 arrayobject.c
*** arrayobject.c 1996/11/28 07:10:25 1.1
--- arrayobject.c 1996/11/29 19:13:11
***************
*** 1612,1641 ****
}
#define ByCopy 1
#define Contiguous 2
! PyObject *array_fromobject(PyObject *op, int type, int min_depth, int max_depth, int flags) {
! PyObject *r;
!
r = NULL;
! if (!PyArray_Check(op) && PyObject_HasAttrString(op, "__array__")) {
PyObject *ap, *arglist;
if (type == PyArray_NOTYPE) {
arglist = Py_BuildValue("()");
} else {
arglist = Py_BuildValue("(c)", type);
}
! ap = PyObject_GetAttrString(op, "__array__");
! r = PyEval_CallObject(ap, arglist);
Py_DECREF(arglist);
!
! Py_XINCREF(r);
! }
!
if (type == PyArray_NOTYPE) {
type = PyArray_ObjectType(op, 0);
}
if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT ||
--- 1612,1646 ----
}
#define ByCopy 1
#define Contiguous 2
! PyObject *array_fromobject(PyObject *op_in, int type, int min_depth, int max_depth, int flags) {
! PyObject *r, *op;
!
r = NULL;
! if (!PyArray_Check(op_in) && PyObject_HasAttrString(op_in, "__array__")) {
! /* __array__(self, type=None) method interface
! for getting an object as an array. */
PyObject *ap, *arglist;
if (type == PyArray_NOTYPE) {
arglist = Py_BuildValue("()");
} else {
arglist = Py_BuildValue("(c)", type);
}
! ap = PyObject_GetAttrString(op_in, "__array__");
! op = PyEval_CallObject(ap, arglist);
! Py_DECREF(ap);
Py_DECREF(arglist);
! if (op == NULL) return NULL;
! } else {
! op = op_in;
! Py_INCREF(op);
! }
!
if (type == PyArray_NOTYPE) {
type = PyArray_ObjectType(op, 0);
}
if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT ||
***************
*** 1648,1673 ****
Py_INCREF(op);
r = op;
}
} else {
if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num;
! if ((r = PyArray_Cast((PyArrayObject *)op, type)) == NULL) return NULL;
}
} else {
r = Array_FromSequence(op, type, min_depth,max_depth);
! if (r == NULL) {
! if (min_depth <= 0) {
! PyErr_Clear();
! r = PyArray_FromScalar(op, type);
! } else {
! return NULL;
! }
}
}
if (r == NULL) return NULL;
! if(!PyArray_Check(r)) {
PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array");
return NULL;
}
if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) {
Py_DECREF(r);
--- 1653,1677 ----
Py_INCREF(op);
r = op;
}
} else {
if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num;
! r = PyArray_Cast((PyArrayObject *)op, type);
}
} else {
r = Array_FromSequence(op, type, min_depth,max_depth);
! if (r == NULL && min_depth <= 0) {
! PyErr_Clear();
! r = PyArray_FromScalar(op, type);
}
}
+ /* finished with op */
+ Py_DECREF(op);
+
if (r == NULL) return NULL;
! if (!PyArray_Check(r)) {
PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array");
return NULL;
}
if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) {
Py_DECREF(r);
rcsdiff: RCS/multiarray.c,v: No such file or directory
===================================================================
RCS file: Numeric/Numeric.py,v
retrieving revision 1.1
diff -c5 -r1.1 Numeric/Numeric.py
*** Numeric/Numeric.py 1996/11/29 20:24:40 1.1
--- Numeric/Numeric.py 1996/11/29 20:25:12
***************
*** 31,40 ****
--- 31,41 ----
return m.astype(typecode)
else:
return m
#Include some functions straight from multiarray
+ asarray = multiarray.asarray
array = multiarray.array
zeros = multiarray.zeros
fromstring = multiarray.fromstring
take = multiarray.take
reshape = multiarray.reshape
***************
*** 235,251 ****
def nonzero(a):
"""Return the indices of the elements of a which are not zero, a must be 1d
"""
return repeat(arange(len(a)), not_equal(a, 0))
! def asarray(a, typecode=None):
! if type(a) == arraytype and (typecode == None or typecode == a.typecode()):
! return a
! elif typecode == None:
! return array(a)
! else:
! return array(a, typecode)
#Move this into C to do it right!
def shape(a):
return asarray(a).shape
--- 236,252 ----
def nonzero(a):
"""Return the indices of the elements of a which are not zero, a must be 1d
"""
return repeat(arange(len(a)), not_equal(a, 0))
! ## def asarray(a, typecode=None):
! ## if type(a) == arraytype and (typecode == None or typecode == a.typecode()):
! ## return a
! ## elif typecode == None:
! ## return array(a)
! ## else:
! ## return array(a, typecode)
#Move this into C to do it right!
def shape(a):
return asarray(a).shape
=================
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================