From phil@geog.ubc.ca Fri Nov 1 02:22:24 1996 From: phil@geog.ubc.ca (Phil Austin) Date: Thu, 31 Oct 1996 18:22:24 -0800 (PST) Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: <199610311925.OAA28124@python.org> References: <9610311758.AA16110@goldilocks.LCS.MIT.EDU> <199610311925.OAA28124@python.org> Message-ID: <199611010222.SAA06674@coot.geog.ubc.ca> >>>>> "GF" == Geoffrey Furnish writes: GF> The PLplot module supports output to a Tk graphics widget GF> (among a multitude of other output drivers), which can be GF> trivially integrated into Python/Tk, and we have that in GF> operation here at LLNL now. There is also the beginnings of a GF> Python/Tk "megawidget" for the plframe graphics widget, which GF> provides a variety of conveniences for scientific plotting, GF> including saving to a any of a variety of supported graphics GF> formats (including postscript and ppm), as well as GF> manipulation of the plot orientation, printing the currently GF> visible view. etc. There is also a binding available for GF> zoom support, and "zoom frame list walking", with panning. This is great news. One of my own projects will be positioning Python as a Matlab replacement for the 7000 undergrads in our faculty of science. Being able to do graphics under Windows95 is a major prerequisite for this. My other hope is that the door is still open on the future addtion of Splus-style syntactic sugar allowing subscripts that are array masks. One sure way to impress a new user in Splus is to show them how to histogram part of a 3-d array with: hist(array[array > 40. & array < 50.]) I can also commit to providing on-going documentation and support for the netcdf module in conjunction with Bill Noon. Phil Austin INTERNET: phil@geog.ubc.ca (604) 822-2175 FAX: (604) 822-6150 http://www.geog.ubc.ca/~phil Associate Professor Atmospheric Sciences Programme Geography #217 University of British Columbia 1984 W Mall Vancouver, BC V6T 1Z2 CANADA ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From furnish@laura.llnl.gov Fri Nov 1 03:43:45 1996 From: furnish@laura.llnl.gov (Geoffrey Furnish) Date: Thu, 31 Oct 1996 19:43:45 -0800 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: <199611010222.SAA06674@coot.geog.ubc.ca> References: <9610311758.AA16110@goldilocks.LCS.MIT.EDU> <199610311925.OAA28124@python.org> <199611010222.SAA06674@coot.geog.ubc.ca> Message-ID: <199611010345.WAA29560@python.org> Phil Austin writes: > >>>>> "GF" == Geoffrey Furnish writes: > GF> The PLplot module supports output to a Tk graphics widget > GF> (among a multitude of other output drivers), which can be > GF> trivially integrated into Python/Tk, and we have that in > GF> operation here at LLNL now. There is also the beginnings of a > GF> Python/Tk "megawidget" for the plframe graphics widget, which > GF> provides a variety of conveniences for scientific plotting, > GF> including saving to a any of a variety of supported graphics > GF> formats (including postscript and ppm), as well as > GF> manipulation of the plot orientation, printing the currently > GF> visible view. etc. There is also a binding available for > GF> zoom support, and "zoom frame list walking", with panning. > > > This is great news. One of my own projects will be positioning Python > as a Matlab replacement for the 7000 undergrads in our faculty of > science. Being able to do graphics under Windows95 is a major > prerequisite for this. I will have to back pedal a little. The Tk plframe widget currently uses the X11 drawing widget, which is not available on other-than-unix. There has, however, been a patch submitted for a driver implemented only in terms of Tk library features, which is cross platform. This widget has not been fully integrated into PLplot yet, however, and although I can say "it will happen", I cannot be precise about timing. -- Geoffrey Furnish email: furnish@llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-6172 ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From fredrik_lundh@ivab.se Fri Nov 1 08:17:30 1996 From: fredrik_lundh@ivab.se (Fredrik Lundh) Date: Fri, 1 Nov 1996 09:17:30 +0100 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: (pas@lems.brown.edu) Message-ID: <9611010817.AA09684@arnold.image.ivab.se> > Sounds good, especially 3. I think light weight Py/Tk widgets would be > the way to go. FWIW, PIL 0.2final will include a "Path" object (2D polyline/beziers, and some path processing primitives), and a first stab at a light- weight plotting package based on these primitives. To avoid some of the worst performance bottlenecks in Tk, output rendering is currently done via raster images (or directly to PostScript). Future versions will probably include patches allowing you to place a path directly on a canvas. Now, could I just find some decent contouring code somewhere... Regards /F ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Fri Nov 1 14:44:03 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Fri, 1 Nov 96 15:44:03 +0100 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: (message from Johann Hibschman on Thu, 31 Oct 1996 12:47:57 -0800 (PST)) Message-ID: <199611011544.QAA19758@ibs.ibs.fr> > Meanwhile, I'm keeping myself busy fending off periodic assaults from > friends saying that I should just use either IDL or C and be done with it. > My CS friends think perl is the answer to all problems. IDL's great, but If you want yet another opinion: use Python and be done with it ;-) Seriously, all such suggestions make sense only for a specific set of problems to be solved. There is no one and only tool that will make everyone happy. Python + NumPy + the occasional module written in C have proven to be the best solution for my needs, but your mileage may vary. > would take a zillion lines of code to do anything interesting. I'm just > not understood. Sigh. :) You should consider yourself lucky. I have to fight against people who think that *Fortran* is the sole path to hapiness! ;-) > (I am somewhat disturbed by reports that perl is several times faster > than python, though.) When seeing such a report, your immediate question should be: for what? Languages like Python and Perl tend to be fast for the purposes that are supported by special low-level modules (written in C), but can be slow for anything else. For array-based number crunching, for example, Python's overall speed is almost irrelevant; all that counts is the speed of the NumPy extension. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Fri Nov 1 14:45:51 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Fri, 1 Nov 96 15:45:51 +0100 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: (pas@lems.brown.edu) Message-ID: <199611011546.QAA19762@ibs.ibs.fr> > PLplot is a bear to compile. I've had a fair amount of experience working > with large software projects, and getting PLplot to the point where I > could use it, in or out of python, took a significant amount of work. Way > too much, in my opinion. And it even uses gnu configure. I have to agree. I tried PLplot a few months ago, but it took forever just to get everything compiled, and the prospect of having to do this repeatedly (new machines, new versions) made be prefer to live without it. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Fri Nov 1 14:51:09 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Fri, 1 Nov 96 15:51:09 +0100 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: <199611010222.SAA06674@coot.geog.ubc.ca> (message from Phil Austin on Thu, 31 Oct 1996 18:22:24 -0800 (PST)) Message-ID: <199611011551.QAA19768@ibs.ibs.fr> > My other hope is that the door is still open on the future addtion of > Splus-style syntactic sugar allowing subscripts that are array masks. > One sure way to impress a new user in Splus is to show them how to > histogram part of a 3-d array with: > > hist(array[array > 40. & array < 50.]) What is that supposed to mean for a 3-d array? Or even for 2-d? Anyway, adding syntactic sugar like this can even be done by writing a Python wrapper for arrays, with no changes to NumPy. Even if that may be too inefficient for real applications, it is a great way to test new proposals such as yours. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From phil@geog.ubc.ca Fri Nov 1 15:42:44 1996 From: phil@geog.ubc.ca (Phil Austin) Date: Fri, 1 Nov 1996 07:42:44 -0800 (PST) Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: <199611011551.QAA19768@ibs.ibs.fr> References: <199611010222.SAA06674@coot.geog.ubc.ca> <199611011551.QAA19768@ibs.ibs.fr> Message-ID: <199611011542.HAA07159@coot.geog.ubc.ca> >> hist(array[array > 40. & array < 50.]) >>>>> "KH" == Konrad Hinsen writes: KH> What is that supposed to mean for a 3-d array? Or even for KH> 2-d? The default in Splus is to flatten the comparison when it appears as the sole subscript. If you want to work with individual indexes, you'd add empty indexes as in python (although there're no pseudo indexes in splus). I wonder if anyone else on the list is familiar with Splus besides David Ascher? I've used it since 1982 (it was the instructional language for Washington's grad stats courses), and its compact indexing notation is unambiguous and very intuitive (to me). Perhaps I'll implement some of its features as Konrad suggests (possibly as soon as my 2 year-old starts elementary school). Phil ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From da@maigret.cog.brown.edu Fri Nov 1 16:22:52 1996 From: da@maigret.cog.brown.edu (David Ascher) Date: Fri, 1 Nov 1996 11:22:52 -0500 (EST) Subject: [PYTHON MATRIX-SIG] Calling all extensions... Message-ID: <199611011622.LAA16382@maigret> I'll be giving a talk at the Python Workshop on Monday. One of the things I'll be spending a couple of minutes on because Jim can't go is a status report of the Numeric extensions. If you'd like me to mention that you're working on a specific extension, please let me know ASAP. The ones I have on my overhead as of now are: LAPACK, FFTPACK, RANLIB, NetCDF, PLPLOT, OpenGL, GIST (although the Gist extension is the subject of its own presentation by its authors) and Konrad's extensions. Anything else that should be mentioned? --david ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From furnish@laura.llnl.gov Fri Nov 1 17:56:33 1996 From: furnish@laura.llnl.gov (Geoffrey Furnish) Date: Fri, 1 Nov 1996 09:56:33 -0800 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: <199611011546.QAA19762@ibs.ibs.fr> References: <199611011546.QAA19762@ibs.ibs.fr> Message-ID: <199611011758.MAA01364@python.org> Konrad Hinsen writes: > > PLplot is a bear to compile. I've had a fair amount of experience working > > with large software projects, and getting PLplot to the point where I > > could use it, in or out of python, took a significant amount of work. Way > > too much, in my opinion. And it even uses gnu configure. > > I have to agree. I tried PLplot a few months ago, but it took forever > just to get everything compiled, and the prospect of having to > do this repeatedly (new machines, new versions) made be prefer to > live without it. With all this public bashing of the PLplot configuration support, I think I have to point out some facts which will be missed by people who have never actually tried it and are going to be very disuaded by statements like this, which I personally regard as fairly exaggerated. Here's why: First, it is certainly unarguable that PLplot's configuration script (GNU autoconf based), has historically had a lot of trouble correctly identifying every single part of a user's supporting environment during autoconfiguration. And I won't deny the frustration many have felt, including the two quoted above. However, you have to understand clearly that the /worst possible case/ with PLplot is exactly the /standard/ case with Python and NumPy itself. In particular, PLplot's configure works very hard to automatically detect the presence and version number of all the supporting libraries the user may have. X11, Tcl/Tk, Python, and others, all fall into this category. And it is true, I admit, that PLplot's configure has, on some systems, had trouble sorting it all out. Note, however, that both Python and NumPy, make absolutely no effort whatsoever to "auto detect" this kind of stuff. For instance, say you are building python on a system on which you have already installed Tcl/Tk. Is Tkinter automatically configured? No. No attempt whatsoever is made to find your Tcl/Tk, and edit your Setup to turn on the compilation of the Tkinter module. You have to do it by hand, editing Setup, changing the lines for the tkinter module, filling in correct path information, etc. You have to edit _tkinter.c by hand to patch in package initializations which Python doesn't know about, etc. When you install NumPy, does it look to see if you have a precompiled FFTPACK sitting around? No. The instructions suggest you use the C version provided, and if you want to use some other version, you have to edit the setup by hand. No big deal. Nobody complains about this, and neither do I. And that situation, is the worst possible situation with PLplot. If autoconf doesn't do all the magic for you, you get to do it by hand. If you try to compile PLplot against Tcl 6.7 and Tk 3.6, and you have trouble, the situation is no different than if you did the same blunder with Python's Tkinter. Etc. What PLplot does do, that neither Python nor NumPy do, is try very hard to sort out what you have on your system, and autoconfigure everything possible. For large numbers of people (read, thouusands) this has worked adequately. For some, it has failed miserably. For those unlucky souls, the worst thing that happens is they have to make a barebones makefile, and edit it by hand, which is exactly what /everyone/ has to do with Python and NumPy themselves. And, moreover, I have recently corrected dozens of autodetection bugs in PLplot's configure, so that the number of "unlucky souls" will be substantially lower than it was before. Anyone who /still/ has major problems with the PLplot autoconf situation is respectfully invited to air their grievances on the PLplot mailing list, where there are plenty of people who could help them with valuable advice. -- Geoffrey Furnish email: furnish@llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-6172 ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From drh@cse.unl.edu Fri Nov 1 18:31:32 1996 From: drh@cse.unl.edu (Doug Heisterkamp) Date: Fri, 1 Nov 1996 12:31:32 -0600 (CST) Subject: [PYTHON MATRIX-SIG] Calling all extensions... Message-ID: <199611011831.MAA16560@cse.unl.edu> > > I'll be giving a talk at the Python Workshop on Monday. One of the > things I'll be spending a couple of minutes on because Jim can't go is a > status report of the Numeric extensions. If you'd like me to mention that > you're working on a specific extension, please let me know ASAP. The > ones I have on my overhead as of now are: LAPACK, FFTPACK, RANLIB, > NetCDF, PLPLOT, OpenGL, GIST (although the Gist extension is the subject > of its own presentation by its authors) and Konrad's extensions. > Anything else that should be mentioned? > I have a low-level access module to ODRPACK. One thing to note is that I have it setup so I can use Python, C, or Fortran functions in ODRPACK. I'm in the process of generalizing this into a separate module so I can use it with other libraries. I will put it up for ftp soon (probably sometime next week). ODRPACK itself is available from netlib. Doug Heisterkamp drh@cse.unl.edu PS: A short blurb on ODRPACK from it's readme file: *** INTRODUCTION ODRPACK is a portable collection of Fortran subprograms for fitting a model to data. It is designed primarily for instances when the explanatory as well as the response variables have significant errors, implementing a highly efficient algorithm for solving the weighted orthogonal distance regression problem, i.e., for minimizing the sum of the squares of the weighted orthogonal distances between each data point and the curve described by the model equation. It can also be used to solve the ordinary least squares problem where all of the errors are attributed to the observations of the dependent variable. File CHANGES identifies how the current version of ODRPACK differs from earlier releases. ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From tom.schwaller@linux-magazin.de Sat Nov 2 02:04:20 1996 From: tom.schwaller@linux-magazin.de (Tom Schwaller) Date: Sat, 02 Nov 1996 03:04:20 +0100 Subject: [PYTHON MATRIX-SIG] Calling all extensions... References: <199611011831.MAA16560@cse.unl.edu> Message-ID: <327AAC24.4EEAE11E@linux-magazin.de> Doug Heisterkamp wrote: > I have a low-level access module to ODRPACK. One thing to note is that I > have it setup so I can use Python, C, or Fortran functions in ODRPACK. I'm > in the process of generalizing this into a separate module so I can use it > with other libraries. I will put it up for ftp soon (probably sometime > next week). ODRPACK itself is available from netlib. > Funny enough, we redo all the work done many times before (Matlab, Octave, Scilab and the last in this series http://zaphod.uni-muenster.de:80/xi/ Your general approach seems interesting. We should work much more in that direction. Btw, has anybody written a general module to dynamicaly load numericl procedures after declaring their argument types. There is a module in the python distribution, but it's very rudimentary. I do not remeber the exact form in Scilab, but in Python it should work like: C-Function: double func(int n, double* a, int r, double s) f=dynload(func, 'd', 'i', 'D', 'i', 'd') (or f=funcload('diDid') , or f=funcload('d', 'iDid')) res = f(10, arange(10), 3, 5.3) Some months ago I also saw a Tcl-extenion for compiling simple C-Cyntax like expressions (just Tcl strings) to C-Functions on the fly.. It works like the example above, but instead of having the C-function at the beginning, you declare it with something like: f=declarefunc('d', 'iDid', 'nars', "\ int j=0; double res=0; for (j=0; j Hi, When building the latest release (Linux 2.0.20, gcc 2.7.0), I get lots of the following errors when compiling NumPy ./multiarraymodule.c:1003: warning: comparison between pointer and integer ./multiarraymodule.c: In function `array_binarysearch': I mean to say lots of lines in multiarray give the same error. Also, when does a NumPy arrive which compiles with python 1.4? One more -- has anybody written any (fairly flexible) nonlinear-least-squares package (esp. where the proper bookkeeping is done so that one can selectively fix parameters durin fit passes)? JT ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From da@maigret.cog.brown.edu Sat Nov 2 15:55:34 1996 From: da@maigret.cog.brown.edu (David Ascher) Date: Sat, 2 Nov 1996 10:55:34 -0500 (EST) Subject: [PYTHON MATRIX-SIG] errors and questions in building latest release In-Reply-To: <199611020255.VAA17969@studbolt.mit.edu> from "Jeffrey Templon" at Nov 1, 96 09:55:49 pm Message-ID: <199611021555.KAA17392@maigret> > ./multiarraymodule.c:1003: warning: comparison between pointer and integer > ./multiarraymodule.c: In function `array_binarysearch': > > I mean to say lots of lines in multiarray give the same error. Note that these are warnings, not errors == it'll work just fine. > Also, when does a NumPy arrive which compiles with python 1.4? Don't know, but FYI, to get the latest a5 to compile with 1.4, all that is needed is changing Ellipses to Ellipsis in arrayobject.c. That doesn't fix the other bugs in 1.0a5, but at least it works. --david ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From jbaddor@phy.ulaval.ca Sun Nov 3 15:52:27 1996 From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR) Date: Sun, 3 Nov 1996 10:52:27 -0500 (EST) Subject: [PYTHON MATRIX-SIG] LinAlg.py vs LinearAlgebra.py Message-ID: Hi, Does the new LinearAlgebra.py package require the matrix to be transposed before and after treatment, like the old Linalg ? Jean-Bernard ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Sun Nov 3 16:45:06 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Sun, 3 Nov 96 17:45:06 +0100 Subject: [PYTHON MATRIX-SIG] Data analysis package question In-Reply-To: <199611011858.TAA20155@ibs.ibs.fr> (message from Geoffrey Furnish on Fri, 1 Nov 1996 09:56:33 -0800) Message-ID: <199611031745.SAA23261@ibs.ibs.fr> > However, you have to understand clearly that the /worst possible case/ > with PLplot is exactly the /standard/ case with Python and NumPy > itself. In particular, PLplot's configure works very hard to > automatically detect the presence and version number of all the > supporting libraries the user may have. X11, Tcl/Tk, Python, and > others, all fall into this category. And it is true, I admit, that > PLplot's configure has, on some systems, had trouble sorting it all > out. Well, I prefer to be told to do it by hand rather than have to figure out from an unclear error message that some program was trying to be smart and failed. Besides, it wasn't just this kind of problem. PLplot didn't find one of its own header files, and I even had to modify some of the source files because they wouldn't get along with the SGI system headers. > Note, however, that both Python and NumPy, make absolutely no effort > whatsoever to "auto detect" this kind of stuff. For instance, say you > are building python on a system on which you have already installed > Tcl/Tk. Is Tkinter automatically configured? No. No attempt > whatsoever is made to find your Tcl/Tk, and edit your Setup to turn on > the compilation of the Tkinter module. You have to do it by hand, With Python, I follow a series of documented steps and get a working installation within minutes. With PLplot, it took me two days before I could run the test examples (of course I didn't work full time on it). That's the bottom line. > And that situation, is the worst possible situation with PLplot. If > autoconf doesn't do all the magic for you, you get to do it by hand. Fine, but that ought to be pointed out clearly. > Anyone who /still/ has major problems with the PLplot autoconf > situation is respectfully invited to air their grievances on the > PLplot mailing list, where there are plenty of people who could help > them with valuable advice. Well, I'll try the latest version as soon as I find some free minutes... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Sun Nov 3 17:09:35 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Sun, 3 Nov 96 18:09:35 +0100 Subject: [PYTHON MATRIX-SIG] LinAlg.py vs LinearAlgebra.py In-Reply-To: (message from Jean-Bernard ADDOR on Sun, 3 Nov 1996 10:52:27 -0500 (EST)) Message-ID: <199611031810.TAA23321@ibs.ibs.fr> > Does the new LinearAlgebra.py package require the matrix to be transposed > before and after treatment, like the old Linalg ? >From the point of view of the user, neither the old nor the new package should require any transposition. And since internally they both use LAPACK, there shouldn't be any difference either. If you ever transposed matrices while working with LinearAlgebra, I suppose your interpretation of matrix indices is different from the one adopted in NumPy! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Sun Nov 3 17:30:36 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Sun, 3 Nov 96 18:30:36 +0100 Subject: [PYTHON MATRIX-SIG] Calling all extensions... In-Reply-To: <327AAC24.4EEAE11E@linux-magazin.de> (message from Tom Schwaller on Sat, 02 Nov 1996 03:04:20 +0100) Message-ID: <199611031831.TAA23361@ibs.ibs.fr> > This is a dirty, not so Python-like solution, but is there a better one > for scientific computations. For heavy computations you need compiled > functions, which you can > generate on the fly, evaluating a interpreted function 10000 time for a > computation > is to slow. Any comments on that. Good question. A while ago I tried the approach of implementing a data type "numerical expression" that would pretend to be a numerical data type and could therefore be passed to any Python code doing numerical calculations. The result would be a compiled version of that code (still using a bytecode interpreter, but with typed data objects and no memory allocation overhead during evaluation) that would be a callable Python object. It never got finished, partly due to time restrictions. However, I realised that this approach had a few problems: 1) The speedup would be significant only for complicated code or loops, but since Python's branching and looping constructs could not be used, the idea of profitably compiling ordinary Python code in this way is probably illusionary. 2) Python's implementation of comparisons made using this extension really messy for nontrivial application. 3) For really CPU intensive calculations the stack-based evaluation engine was still too slow. Nevertheless, I still think that some compilation method not requiring to run the C compiler would be nice as an intermediate step. Not everyone knows C, and not all systems support dynamic libraries, without which the idea of just-in-time compilation doesn't make sense. On the other hand, an easy way to interface to existing C and Fortran functions would obviously be nice as way. "Easy" would ideally mean an approach that can be used by someone who does not know C and/or Fortran. Unfortunately I don't have any good solution at hand... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Mon Nov 4 09:34:19 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Mon, 4 Nov 96 10:34:19 +0100 Subject: [PYTHON MATRIX-SIG] C API questions Message-ID: <199611041034.LAA25034@ibs.ibs.fr> For the first time I have to use the C API to the array objects in a non-trivial way, and I'd like to profit a bit from the expertise of those who have done it before. I want to provide a C interface to functions written in Python, which means I have to construct array objects referring to a given C array of arbitrary origin. This raises a few questions about memory allocation. and reference counting. 1) How should I construct the array object to make sure that Python will not try to allocate or free memory for the actual data? 2) Should my code simply pass array objects to the Python code and then forget about them, or should it increase the reference count before the call and decrement it afterwards? I suppose that if everything goes well that shouldn't make a difference. 3) What should I do to prevent the Python code from keeping a reference to the arrays even after returning (e.g. by putting it into a global list)? Is there any legitimate reason to do this? Should I check the reference count after the call and raise an exception if it is too high? I must somehow make sure that the temporary array object will be deleted, since I can't guarantee for the existence of the data part after my function finishes its work. 4) Is there any other problem I should watch out for? Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From drh@cse.unl.edu Mon Nov 4 20:03:37 1996 From: drh@cse.unl.edu (Doug Heisterkamp) Date: Mon, 4 Nov 1996 14:03:37 -0600 (CST) Subject: [PYTHON MATRIX-SIG] Calling all extensions... In-Reply-To: <327AAC24.4EEAE11E@linux-magazin.de> from "Tom Schwaller" at Nov 2, 96 03:04:20 am Message-ID: <199611042003.OAA19994@cse.unl.edu> > > > Your general approach seems interesting. We should work much more in > that direction. > > Btw, has anybody written a general module to dynamicaly load numericl > procedures > after declaring their argument types. There is a module in the python > distribution, > but it's very rudimentary. I assume you mean dlmodule. I have not look at that module before -- I'd assume it was Sun specific. Is it portable and is there documentation for it? > > I do not remeber the exact form in Scilab, but in Python it should work > like: > > C-Function: double func(int n, double* a, int r, double s) > > f=dynload(func, 'd', 'i', 'D', 'i', 'd') (or f=funcload('diDid') , or > f=funcload('d', 'iDid')) > res = f(10, arange(10), 3, 5.3) > I think something like this is possible. I would prefer separating the return type from the parameter types and making the return type optional with a default of returning None. I would also like an optional argument for the library which holds the function. The default libraries list could be kept in the dynamic load module. > Some months ago I also saw a Tcl-extenion for compiling simple > C-Cyntax like expressions (just Tcl strings) to C-Functions on the fly.. > > > This is a dirty, not so Python-like solution, but is there a better one > for scientific computations. For heavy computations you need compiled > functions, which you can > generate on the fly, evaluating a interpreted function 10000 time for a > computation > is to slow. Any comments on that. Generating compiled functions on the fly can be done right now if you have a set interface. Doing it portably could be difficult. But it can be done by (1) writing the text to a file, (2) executing a makefile that (a) compiles the text file, (b) places it into a library, (c) relinks the dynamically loaded module that holds the interface function with the library, (d) copy the module to python's sharedmodule directory, (3) reload the module. If we can get a dynload module working then the restriction on having a set interface is gone, and also the restriction of using only a certain number of functions at one time is gone (the interface module has a static number of function interfaces). As to interpreted functions being to slow for heavy computations, I think it will depend on the problem. But in general, I think it will be true. I'm hopeful that if the function is using the array operations then it will run in a reasonable amount of time. Even if it does not, there is always late friday afternoons. Where letting the machine run over the weekend is much better than staying into the evening and writing the C functions. > > What about sparse matrices. I remeber we had a discussion on that a very > long time ago. > I propose taking the sparsepack library and melting it with NumPy. Which sparse matrix library do you use? I don't need to use sparse matrices at the moment, but I would be willing to help someone else in setting up the interface with NumPy. Doug Heisterkamp drh@cse.unl.edu ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From drh@cse.unl.edu Mon Nov 4 20:50:52 1996 From: drh@cse.unl.edu (Doug Heisterkamp) Date: Mon, 4 Nov 1996 14:50:52 -0600 (CST) Subject: [PYTHON MATRIX-SIG] C API questions In-Reply-To: <199611041034.LAA25034@ibs.ibs.fr> from "Konrad Hinsen" at Nov 4, 96 10:34:19 am Message-ID: <199611042050.OAA25684@cse.unl.edu> > For the first time I have to use the C API to the array objects in a > non-trivial way, and I'd like to profit a bit from the expertise of > those who have done it before. > > I want to provide a C interface to functions written in Python, which > means I have to construct array objects referring to a given C array > of arbitrary origin. This raises a few questions about memory allocation. > and reference counting. > I'm am not an expert on the C API, but I am doing similar things. I'll relate what I'm currently doing in the hope that I'll be corrected if it is not right. > 1) How should I construct the array object to make sure that Python > will not try to allocate or free memory for the actual data? I am using PyArray_FromDimsAndData. If data is passed into this function, then the flag OWN_DATA is not set. In array_dealloc, if OWN_DATA is not set, then the data pointer of the array object is not freed. > > 2) Should my code simply pass array objects to the Python code and > then forget about them, or should it increase the reference count > before the call and decrement it afterwards? I suppose that if > everything goes well that shouldn't make a difference. I decrement the reference to the array object after the python function call. I do not increment it before the function call. The array object reference count was incremented in the PyArray_FromDimsAndData function by its call to PyObject_NEW (I think--not very sure on some of this stuff). > > 3) What should I do to prevent the Python code from keeping a reference > to the arrays even after returning (e.g. by putting it into a > global list)? Is there any legitimate reason to do this? Should I > check the reference count after the call and raise an exception if > it is too high? I must somehow make sure that the temporary array > object will be deleted, since I can't guarantee for the existence > of the data part after my function finishes its work. > I have not done this, but I should. Checking the reference count and raising an exception should work in most cases. > 4) Is there any other problem I should watch out for? Just the typical stuff: remember to DECREF the argument list if you created it with Py_BuildValue, remember the DECREF the results returned from the python function when you are done with them, and if calling from a Fortran function remember to reverse the indexing order when building the array object. Doug Heisterkamp drh@cse.unl.edu ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From tom.schwaller@linux-magazin.de Mon Nov 4 23:59:55 1996 From: tom.schwaller@linux-magazin.de (Tom Schwaller) Date: Tue, 05 Nov 1996 00:59:55 +0100 Subject: [PYTHON MATRIX-SIG] Calling all extensions and Sparselib References: <199611042003.OAA19994@cse.unl.edu> Message-ID: <327E837B.5FFC6C57@linux-magazin.de>  > > I assume you mean dlmodule. I have not look at that module before -- I'd > assume it was Sun specific. Is it portable and is there documentation > for it? Hmm, the dl interface is implemented on all architectures I know. When you have dynamic loading of modules running, then the dlmodule should work too.. > > > > I do not remeber the exact form in Scilab, but in Python it should work > > like: > > > > C-Function: double func(int n, double* a, int r, double s) > > > > f=dynload(func, 'd', 'i', 'D', 'i', 'd') (or f=funcload('diDid') , or > > f=funcload('d', 'iDid')) > > res = f(10, arange(10), 3, 5.3) > > > > I think something like this is possible. I would prefer separating the > return type from the parameter types and making the return type optional > with a default of returning None. I would also like an optional argument > for the library which holds the function. The default libraries list > could be kept in the dynamic load module. Good point.  > > Generating compiled functions on the fly can be done right now if you > have a set interface. Doing it portably could be difficult. But it > can be done by (1) writing the text to a file, (2) executing a makefile > that (a) compiles the text file, (b) places it into a library, (c) relinks > the dynamically loaded module that holds the interface function with > the library, (d) copy the module to python's sharedmodule directory, > (3) reload the module. Thats exactly the approcha of the mentioned Tcl module, whichhhhh works quite well.. > > As to interpreted functions being to slow for heavy computations, I think > it will depend on the problem. But in general, I think it will be true. > I'm hopeful that if the function is using the array operations then > it will run in a reasonable amount of time. Even if it does not, there > is always late friday afternoons. Where letting the machine run over the > weekend is much better than staying into the evening and writing the > C functions. > I had some tests 2 years ago with PDE stuff where I used Python to plug in functions (written very fast with Python) and the results where absolutely disapointing. Also Tests with nonlinear optimisation stuff were very bad.. > > What about sparse matrices. I remeber we had a discussion on that a very > > long time ago. > > I propose taking the sparsepack library and melting it with NumPy. > > Which sparse matrix library do you use? I don't need to use sparse > matrices at the moment, but I would be willing to help someone else > in setting up the interface with NumPy. Im not exactly shure which one I one discoverd on the net, but I think it's http://math.nist.gov/spblas/ There are also C++ Wrapper code at http://math.nist.gov/sparselib++ Very important for our purpose is http://math.nist.gov/acmd/Staff/KRemington/spblasc/spblasc.html http://www.cs.umn.edu/research/darpa/p_sparslib/psp-abs.html is also very interesting. I propose to use the first code (Sparselib) and here"s another link http://math.nist.gov/iml++ which one should consider for preconditioning stuff.. Hope this is interesting stuff, if you do not know it. Tom ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From phil@geog.ubc.ca Tue Nov 5 00:59:43 1996 From: phil@geog.ubc.ca (Phil Austin) Date: Mon, 4 Nov 1996 16:59:43 -0800 (PST) Subject: [PYTHON MATRIX-SIG] Calling all extensions and Sparselib In-Reply-To: <327E837B.5FFC6C57@linux-magazin.de> References: <199611042003.OAA19994@cse.unl.edu> <327E837B.5FFC6C57@linux-magazin.de> Message-ID: <199611050059.QAA09421@coot.geog.ubc.ca> >>>>> "Tom" == Tom Schwaller writes: >> Which sparse matrix library do you use? I don't need to use sparse >> matrices at the moment, but I would be willing to help someone else >> in setting up the interface with NumPy. Tom> Im not exactly shure which one I one discoverd on the net, but I think Tom> it's Tom> http://math.nist.gov/spblas/ Also worth a look is the more elaborate:
PETSc - The Portable, Extensible Toolkit for Scientific Computation: sparse pdes Phil ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Tue Nov 5 17:52:29 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 5 Nov 96 18:52:29 +0100 Subject: [PYTHON MATRIX-SIG] C API questions In-Reply-To: <199611042048.OAA25223@cse.unl.edu> (drh@cse.unl.edu) Message-ID: <199611051752.SAA02842@ibs.ibs.fr> > > 1) How should I construct the array object to make sure that Python > > will not try to allocate or free memory for the actual data? > > I am using PyArray_FromDimsAndData. If data is passed into this function, > then the flag OWN_DATA is not set. In array_dealloc, if OWN_DATA is not > set, then the data pointer of the array object is not freed. Seeme easy enough... > I decrement the reference to the array object after the python function > call. I do not increment it before the function call. The array Yes, of course. It starts out at one, as for any newly created object. > I have not done this, but I should. Checking the reference count and > raising an exception should work in most cases. It surely beats a crash due to an access to unallocated memory! Thanks for your comments, it seems clear now what I have to do. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Tue Nov 5 18:45:02 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 5 Nov 96 19:45:02 +0100 Subject: [PYTHON MATRIX-SIG] C API questions In-Reply-To: <9611051825.AA29044@icf.llnl.gov.llnl.gov> (dubois1@llnl.gov) Message-ID: <199611051845.TAA03030@ibs.ibs.fr> > The one part about this discussion I didn't follow was the part about trying > to make sure Python does not keep a reference to your compiled object. That > is probably impossible what with traceback stacks and all. Or maybe I didn't > get it. But the traceback stack does not keep its references forever, I hope. Is there maybe a way to force it to give up a certain reference? > If you do use PyArray_FromDimsAndData you should just follow the normal > Python rules for reference counting in your extensions and since the count > started out at 1 it will never get deleted. But if the pointer points to > something that might not be there later, this is the wrong Python object to > model it, IMHO. But what other choice do I have? I want to define a general C-level interface to some of my code, which might actually be Python code. So I must construct a C wrapper around the Python functions. But a C function can't make any assumptions about the existence of its actual parameters after it returns. It might look like: double *x = malloc(n*sizeof(double)); do_some_calculation(x, n); free(x); with void do_some_calculation(double *x, int n) { PyArrayObject *a = PyArray_FromDimsAndData(.., x, n, ..); /* call Python function working on a */ } In that case I must make sure that no access to the memory pointed to by x can occur after my C wrapper exits. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hugunin@mit.edu Tue Nov 5 19:01:22 1996 From: hugunin@mit.edu (Jim Hugunin) Date: Tue, 5 Nov 1996 14:01:22 -0500 Subject: [PYTHON MATRIX-SIG] C API questions Message-ID: <9611051901.AA10047@goldilocks.LCS.MIT.EDU> > But what other choice do I have? I want to define a general C-level > interface to some of my code, which might actually be Python code. > So I must construct a C wrapper around the Python functions. But a > C function can't make any assumptions about the existence of its > actual parameters after it returns. It might look like: > > double *x = malloc(n*sizeof(double)); > do_some_calculation(x, n); > free(x); > > with > > void do_some_calculation(double *x, int n) > { > PyArrayObject *a = PyArray_FromDimsAndData(.., x, n, ..); > /* call Python function working on a */ > } > > In that case I must make sure that no access to the memory pointed > to by x can occur after my C wrapper exits. My usual solution in this case is to copy data frequently. I assume that you want to modify the array x in-place here? My own approach in these situation has been to change my C API to something like: do_some_calculation(x_in, x_out, n) where x_out is allocated memory to hold the result. then in my python code I do: a = PyArray_FromDims memcpy x_in into a do calculation get back PyArrayObject b memcpy from b's data area into x_out For all of the functions I've used, the impact of these memcpy's on my performance has been too small to measure. I actually have come to like this approach even for straight C code as it makes many memory management issues a little bit more straightforward. -Jim ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Wed Nov 6 09:38:44 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 6 Nov 96 10:38:44 +0100 Subject: [PYTHON MATRIX-SIG] C API questions In-Reply-To: <9611051901.AA10047@goldilocks.LCS.MIT.EDU> (hugunin@mit.edu) Message-ID: <199611060938.KAA05501@ibs.ibs.fr> > My usual solution in this case is to copy data frequently. I assume that > you want to modify the array x in-place here? My own approach in these > situation has been to change my C API to something like: Yes, the data does get modified, but even without modification I wouldn't think of copying, because the arrays can be very large (meaning a few MB, and there are several of them around). But I just got another idea: if before returning I set the array's shape to something corresponding to a zero length array, then there should be no way to access its memory area, right? Ignoring of course buggy C extensions, which I don't care about. To be even more safe, I could change the data pointer to point to some small statically allocated area. Or am I missing something? Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hugunin@mit.edu Wed Nov 6 16:17:09 1996 From: hugunin@mit.edu (Jim Hugunin) Date: Wed, 6 Nov 1996 11:17:09 -0500 Subject: [PYTHON MATRIX-SIG] C API questions Message-ID: <9611061616.AA24982@goldilocks.LCS.MIT.EDU> > > My usual solution in this case is to copy data frequently. I assume that > > you want to modify the array x in-place here? My own approach in these > > situation has been to change my C API to something like: > > Yes, the data does get modified, but even without modification I wouldn't > think of copying, because the arrays can be very large (meaning a few MB, > and there are several of them around). Okay, I'm still not convinced, see my comments below... > But I just got another idea: if before returning I set the array's > shape to something corresponding to a zero length array, then > there should be no way to access its memory area, right? Ignoring > of course buggy C extensions, which I don't care about. To be > even more safe, I could change the data pointer to point to some > small statically allocated area. Or am I missing something? Unfortunately, if the user has taken a slice of the array (or indexed into it if it's more than 1d) they can have an array with a pointer to the data in your original array and there's no way for you to know this (except of course for the original array's reference count). This does appear to be a bit of a problem. I never imagined people would want to call a NumPy function from generic, non-python-aware C. Could you explain a little bit more the reasoning behind this design? At first glance it looks really backwards. I'm also a bit confused as to what you are doing to these arrays in python code. Most operations on arrays involve a factor of 2 or so wasted memory (c = a+b). If you're that concerned about making copies of arrays it's hard for me to imagine what useful things you could do with them in python code (without writing c-like code of the form add(a,b,c) which is not meant for frequent use). -Jim ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Wed Nov 6 16:40:15 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 6 Nov 96 17:40:15 +0100 Subject: [PYTHON MATRIX-SIG] C API questions In-Reply-To: <9611061616.AA24982@goldilocks.LCS.MIT.EDU> (hugunin@mit.edu) Message-ID: <199611061640.RAA09510@ibs.ibs.fr> > Unfortunately, if the user has taken a slice of the array (or indexed into > it if it's more than 1d) they can have an array with a pointer to the data > in your original array and there's no way for you to know this (except of > course for the original array's reference count). It doesn't help me much to know it, I have to be able to prevent it... > This does appear to be a bit of a problem. I never imagined people would > want to call a NumPy function from generic, non-python-aware C. Could you > explain a little bit more the reasoning behind this design? At first The application is force field evaluation for molecular systems. There are a set of "force fields", which are essentially subroutines that take a list of points as input and return a single value. So far no problem, but optionally they also add something to selected elements of one or two arrays passed to them (energy gradients and force constant matrices). There will eventually be many of these subroutines, who may even call one another. And they might be written in Python, C, or Fortran. So what I need is a unified interface (or rather two, one in Python and one in C) that makes it possible to call any of these subroutines without knowing more than its address. The easiest solution would be to pass around array objects; the C functions could easily extract the data from them. But I also want the C interface to be usable from Fortran, and there's the problem. The only data structure known to Fortran is an array, so I *have* to pass C-style arrays in my generic interface. And that causes all the trouble for a force field implemented in Python. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Wed Nov 6 18:53:10 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Wed, 06 Nov 1996 13:53:10 -0500 Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Alpha release of new tutorial for 1.0a5 In-Reply-To: Your message of "Wed, 23 Oct 1996 13:21:12 EDT." <199610231721.NAA22272@maigret> References: <199610231721.NAA22272@maigret> Message-ID: <199611061853.NAA02216@monty> > For most of the python modules, I think TIM would be good enough. > For the scientific extensions which are bound to arrive in droves =), > it'd be nice to have one which allowed math like LaTeX, but also > allowed it to be displayed in HTML form. One short-term solution would be to use conditional inclusion. I believe texinfo supports this -- you can have one version of a paragraph/sentence/word for one back-end (e.g. TeX), and another for the others. I *think*... --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Thu Nov 7 02:46:37 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Wed, 06 Nov 1996 21:46:37 -0500 Subject: [PYTHON MATRIX-SIG] Calling all extensions and Sparselib In-Reply-To: Your message of "Tue, 05 Nov 1996 00:59:55 +0100." <327E837B.5FFC6C57@linux-magazin.de> References: <199611042003.OAA19994@cse.unl.edu> <327E837B.5FFC6C57@linux-magazin.de> Message-ID: <199611070246.VAA04199@monty> > Hmm, the dl interface is implemented on all architectures I know. > When you have dynamic loading of modules running, > then the dlmodule should work too.. Wish that were true :-( Unfortunately, dlmodule only knows how to do it on SunOS/Solaris/SGI and any system that has a compatible implementation. This should probably be rectified -- the code in importdl.c knows how to do dynamic loading on many platforms, and knows very little about Python modules... But for now, you need a sun compatible for dlmodule. Sorry, --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Thu Nov 7 16:15:42 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 7 Nov 96 17:15:42 +0100 Subject: [PYTHON MATRIX-SIG] Memory leak? Message-ID: <199611071615.RAA15486@ibs.ibs.fr> I am beginning to wonder whether there could be a memory leak in some array functions. I am running a rather simple iterative calculation which during each iteration creates a few large matrices (always of exactly the same size) and destroys them in the end (i.e. the reference count ought to become zero). The results of each iteration are accumulated in a list, but they are very small. So the whole program should use a constant amount of memory - but in reality it starts out at 10 MB during the first iteration and then steadily eats up more until it crashes the machine (you can imagine how popular I am among my colleagues after trying this a few times...), which has 128 MB of virtual memory. The highest memory use I have observed is 100 MB, at which point I killed the job. This may of course be a severe case of memory fragmentation, but even then I wouldn't be very happy about it. Any ideas how I could analyze what's wrong? Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hugunin@mit.edu Thu Nov 7 16:40:54 1996 From: hugunin@mit.edu (Jim Hugunin) Date: Thu, 7 Nov 1996 11:40:54 -0500 Subject: [PYTHON MATRIX-SIG] Memory leak? Message-ID: <9611071640.AA11796@goldilocks.LCS.MIT.EDU> > From: Konrad Hinsen > I am beginning to wonder whether there could be a memory leak in some > array functions. I'm fairly convinced that there are some memory leaks in the array functions. I haven't had the time recently to track them down as I'm not currently being bitten by the bug. The very best way to find these bugs is to use the program called Purify. I used this myself on earlier versions of NumPy, but it's gotten much more difficult for me to do this since I moved my primary development work to NT. If you don't have access to Purify, I'd suggest the usual debugging approach of shrinking the problem code to the smallest size you can that still has the bug. You should be able to identify the specific array functions that are casuing problems. At this point I'd be happy to look at the code. -Jim ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From janssen@parc.xerox.com Thu Nov 7 02:34:40 1996 From: janssen@parc.xerox.com (Bill Janssen) Date: Wed, 6 Nov 1996 18:34:40 PST Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Where to go (was Numerical Recipes) In-Reply-To: <199611070219.VAA16356@dolphin.automatrix.com> References: <199610211441.JAA00648@darwin.rsoc.rockwell.com> <199611070219.VAA16356@dolphin.automatrix.com> Message-ID: Excerpts from ext.python: 6-Nov-96 [PYTHON DOC-SIG] Re: [PYTHO.. Skip Montanaro@automatri (826) > (In case it's not clear, I still think migrating to FM is a mistake for > documentation of free > software.) Yes, I tend to agree. Bill ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Wed Nov 6 18:53:10 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Wed, 06 Nov 1996 13:53:10 -0500 Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Alpha release of new tutorial for 1.0a5 In-Reply-To: Your message of "Wed, 23 Oct 1996 13:21:12 EDT." <199610231721.NAA22272@maigret> References: <199610231721.NAA22272@maigret> Message-ID: <199611061853.NAA02216@monty> > For most of the python modules, I think TIM would be good enough. > For the scientific extensions which are bound to arrive in droves =), > it'd be nice to have one which allowed math like LaTeX, but also > allowed it to be displayed in HTML form. One short-term solution would be to use conditional inclusion. I believe texinfo supports this -- you can have one version of a paragraph/sentence/word for one back-end (e.g. TeX), and another for the others. I *think*... --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From skip@calendar.com (Skip Montanaro) Thu Nov 7 02:19:44 1996 From: skip@calendar.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 6 Nov 1996 21:19:44 -0500 (EST) Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Where to go (was Numerical Recipes) In-Reply-To: <199610211441.JAA00648@darwin.rsoc.rockwell.com> References: <199610211441.JAA00648@darwin.rsoc.rockwell.com> Message-ID: <199611070219.VAA16356@dolphin.automatrix.com> Robin Friedrich writes: It should be made clear that the Language Reference and Tutorial are the only docs converted to FrameMaker. It should also be made clear that restricting those docs to FrameMaker makes it doubly difficult to get any help updating them... (In case it's not clear, I still think migrating to FM is a mistake for documentation of free software.) Skip Montanaro | Musi-Cal: http://concerts.calendar.com/ skip@calendar.com | Conference Calendar: http://conferences.calendar.com/ (518)372-5583 | Python: http://www.python.org/ Annoyed with coverage of WWW? http://www.automatrix.com/~skip/diatribe.html ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From janssen@parc.xerox.com Wed Nov 6 20:33:12 1996 From: janssen@parc.xerox.com (Bill Janssen) Date: Wed, 6 Nov 1996 12:33:12 PST Subject: [PYTHON DOC-SIG] Re: [PYTHON MATRIX-SIG] Alpha release of new tutorial for 1.0a5 In-Reply-To: <199611061853.NAA02216@monty> References: <199610231721.NAA22272@maigret> <199611061853.NAA02216@monty> Message-ID: Excerpts from direct: 6-Nov-96 Re: [PYTHON DOC-SIG] Re: [P.. Guido van Rossum@cnri.re (539) > One short-term solution would be to use conditional inclusion. I > believe texinfo supports this -- you can have one version of a > paragraph/sentence/word for one back-end (e.g. TeX), and another for > the others. I *think*... Yes; here's the relevant section from the TeXinfo manual: Using Ordinary TeX Commands =========================== Inside a region delineated by `@iftex' and `@end iftex', you can embed some PlainTeX commands. Info will ignore these commands since they are only in that part of the file which is seen by TeX. You can write the TeX commands as you would write them in a normal TeX file, except that you must replace the `\' used by TeX with an `@'. For example, in the `@titlepage' section of a Texinfo file, you can use the TeX command `@vskip' to format the copyright page. (The `@titlepage' command causes Info to ignore the region automatically, as it does with the `@iftex' command.) However, many features of PlainTeX will not work, as they are overridden by features of Texinfo. You can enter PlainTeX completely, and use `\' in the TeX commands, by delineating a region with the `@tex' and `@end tex' commands. (The `@tex' command also causes Info to ignore the region, like the `@iftex' command.) For example, here is a mathematical expression written in PlainTeX: @tex $$ \chi^2 = \sum_{i=1}^N \left (y_i - (a + b x_i) \over \sigma_i\right)^2 $$ @end tex The output of this example will appear only in a printed manual. If you are reading this in Info, you will not see anything after this paragraph. ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From busby@icf.llnl.gov Fri Nov 8 22:43:24 1996 From: busby@icf.llnl.gov (L. Busby) Date: Fri, 8 Nov 96 14:43:24 PST Subject: [PYTHON MATRIX-SIG] ANNOUNCE: June wishlist patches, new Gist released Message-ID: <9611082243.AA17570@icf.llnl.gov.llnl.gov> I've just posted an announcement to comp.lang.python regarding a set of three large patches to Python-1.4, and a new release of Gist. Numeric users may take special interest. If you do not or cannot see the posting there, please let me know. You can pick up the new stuff at ftp://icf.llnl.gov/pub/python/busby/patches.961108.tgz ftp://icf.llnl.gov/pub/python/busby/pygist-1.2.tgz Lee ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From viennet@ura1507.univ-paris13.fr Sun Nov 10 11:16:55 1996 From: viennet@ura1507.univ-paris13.fr (Emmanuel Viennet) Date: Sun, 10 Nov 1996 11:16:55 GMT Subject: [PYTHON MATRIX-SIG] .flat attribute Message-ID: <199611101116.LAA16555@univ-paris13.fr> It seems that the attribute .flat of an array is usable only if the array is contiguous. (ValueError: flattened indexing only available for contiguous array) This point should be mentionned in the tutorial. Moreover, since a function can't know if the array it will process will be or not contiguous, they are very few cases where .flat is usefull. Or did I miss something ? Emmanuel -- Emmanuel Viennet: LIPN - Institut Galilee - Universite Paris-Nord 93430 Villetaneuse - France ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hugunin@mit.edu Mon Nov 11 20:00:11 1996 From: hugunin@mit.edu (Jim Hugunin) Date: Mon, 11 Nov 1996 15:00:11 -0500 Subject: [PYTHON MATRIX-SIG] .flat attribute Message-ID: <9611112124.AA09565@goldilocks.LCS.MIT.EDU> > It seems that the attribute .flat of an array is usable only if the > array is contiguous. > (ValueError: flattened indexing only available for contiguous array) > > This point should be mentionned in the tutorial. True. The preferred way to get a flattened array is the ravel function. This should always work. > Moreover, since a function can't know if the array it will process will be > or not contiguous, they are very few cases where .flat is usefull. > Or did I miss something ? I start many of my functions with the code: a = array(a, Float) This quarantees that I have a contiguous array of C doubles and that I can modify this array without affecting the arrays held by the caller of the function. The obvious problem with this is that it adds an array copy to every such function. Where this is a problem I don't do it. ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From cgw@pgt.com Wed Nov 13 22:03:33 1996 From: cgw@pgt.com (Charles G Waldman) Date: Wed, 13 Nov 1996 17:03:33 -0500 Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction" Message-ID: <9611132203.AA12113@reactor.pgt> Can anyone explain to me whether the following is a feature or a bug? Python 1.4 (Oct 31 1996) [GCC 2.7.2] Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import ni,Numeric >>> def g(a): ... return a ... >>> x = Numeric.fromfunction(g,[10]) >>> x 0 1 2 3 4 5 6 7 8 9 >>> len(x) 10 >>> >>> >>> def g(a): ... return 5 ... >>> x = Numeric.fromfunction(g,[10]) >>> x 5 >>> len(x) Traceback (innermost last): File "", line 1, in ? TypeError: len() of unsized object >>> >>> >>> def g(a): ... return a*0 + 5 ... >>> x = Numeric.fromfunction(g,[10]) >>> x 5 5 5 5 5 5 5 5 5 5 >>> len(x) 10 >>> This has me quite puzzled. If the initializer function "g" returns a constant, I get a scalar rather than an array, even though I have asked for a shape of [10]. If the function "g" uses its argument (here in a trivial way, multiplied by zero!) then I get the behavior I expect. I first noticed this trying to use "fromfunction" with a random-number generator, to create arrays of random numbers. My code currently works, but it has the strange "multiply by zero" stuck in there, so it looks pretty weird. I'd like to have a better understanding of what's going on. Thanks in advance for any comments! ============================= | Charles G. Waldman | | Princeton Gamma-Tech, Inc.| | cgw@pgt.com | ============================= ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Thu Nov 14 09:23:04 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 14 Nov 96 10:23:04 +0100 Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction" In-Reply-To: <9611132203.AA12113@reactor.pgt> (message from Charles G Waldman on Wed, 13 Nov 1996 17:03:33 -0500) Message-ID: <199611140922.KAA11272@ibs.ibs.fr> > Can anyone explain to me whether the following is a feature or a bug? > > Python 1.4 (Oct 31 1996) [GCC 2.7.2] > Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam > >>> import ni,Numeric > >>> def g(a): > ... return a > ... > >>> x = Numeric.fromfunction(g,[10]) > >>> x > 0 1 2 3 4 5 6 7 8 9 > >>> len(x) > 10 > >>> > >>> > >>> def g(a): > ... return 5 > ... > >>> x = Numeric.fromfunction(g,[10]) > >>> x > 5 Feature, though not a well-known one. fromfunction() works correctly only if the function 1) uses all its arguments 2) consists only of scalar operations defined on arrays (i.e. elementary arithmetic operations and functions from umath). -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From da@maigret.cog.brown.edu Thu Nov 14 15:06:52 1996 From: da@maigret.cog.brown.edu (David Ascher) Date: Thu, 14 Nov 1996 10:06:52 -0500 (EST) Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction" In-Reply-To: <199611140922.KAA11272@ibs.ibs.fr> from "Konrad Hinsen" at Nov 14, 96 10:23:04 am Message-ID: <199611141506.KAA09924@maigret> > Feature, though not a well-known one. fromfunction() works correctly > only if the function > 1) uses all its arguments > 2) consists only of scalar operations defined on arrays (i.e. elementary > arithmetic operations and functions from umath). This is in the tutorial: --- Currently, the function which is called by fromfunction can only use numerical operations and ufuncs on its arguments. It cannot, for example, use if ... else: ... constructs. This is due to the fact that fromfunction does operations "en masse" to speed up evaluation. If you need to fill an array with the result of a function which does not meet these criteria, you can always use a function like: --- I will also add in the next release of the tutorial code from Konrad which removes these limitations (at the expense of speed, alas). --david ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Thu Nov 14 15:42:08 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 14 Nov 96 16:42:08 +0100 Subject: [PYTHON MATRIX-SIG] Odd behavior of "Numeric.fromfunction" In-Reply-To: <199611141506.KAA09924@maigret> (da@maigret.cog.brown.edu) Message-ID: <199611141541.QAA13989@ibs.ibs.fr> > I will also add in the next release of the tutorial code from Konrad > which removes these limitations (at the expense of speed, alas). To the extent of making a general-purpose function worse than a simple nested loop that assigns the elements. So maybe that may be the best solution in many cases. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-76.88.99.28 Institut de Biologie Structurale | Fax: +33-76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From kschalm@geog.ubc.ca Fri Nov 15 21:12:01 1996 From: kschalm@geog.ubc.ca (Kyle Schalm) Date: Fri, 15 Nov 1996 13:12:01 -0800 (PST) Subject: [PYTHON MATRIX-SIG] ncmodule memory fix Message-ID: <199611152112.NAA04134@peacock.geog.ubc.ca> the netcdf module (array version) has a memory fix. you can apply the patch below or download the new version of ncmodule.c from http://www.geog.ubc.ca/~kschalm/ here's the patch (< = new version, > = old version): 1521c1521 < PyObject *list, *m, *t, *ret, *a; /* array taken from arg */ --- > PyObject *list, *m, *t, *a; /* array taken from arg */ 1591,1593c1591 < ret = PyArray_FromDimsAndData(self->ndims, dimlen, array_type, t_data); < if (ret != NULL) ((PyArrayObject *) ret)->flags |= OWN_DATA; < return ret; --- > return PyArray_FromDimsAndData(self->ndims, dimlen, array_type, t_data); --Kyle ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From jhauser@ifm.uni-kiel.de Sat Nov 16 14:16:31 1996 From: jhauser@ifm.uni-kiel.de (janko hauser) Date: Sat, 16 Nov 1996 15:16:31 +0100 Subject: [PYTHON MATRIX-SIG] When 1.06{a,b}? Message-ID: <9611161416.AA03998@caesar.ifm.uni-kiel.de> Hello I want to build a new python with new gistmodule to get commandline-history in the interpreter. Is the new version comming in the near future? If not what are the necessary changes regarding ``ellipsis''? This is the main problem at the moment to build Numpy with Python1.4, isn't it? TIA __Janko ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From viennet@ura1507.univ-paris13.fr Mon Nov 18 12:00:38 1996 From: viennet@ura1507.univ-paris13.fr (Emmanuel Viennet) Date: Mon, 18 Nov 1996 12:00:38 GMT Subject: [PYTHON MATRIX-SIG] Saving HUGE arrays Message-ID: <199611181200.MAA11032@univ-paris13.fr> I am working with huge arrays (size comparable to the total swap space of the machine, ie ~ 100Mo). Currently, we write theses arrays to disk using the following procedure: fp.write( some_header_info ) fp.write( arr.tostring() ) or in some case: fp.write( arr.byteswapped().tostring() ) This works but duplicates the array data to get a string. I know Python strings are immutables, so sharing data between a string and an array is prohibited. Should I write a C simple function write_array( arr, fp, byte_order ) directly writing the data to a file, swapping bytes if requested, or is there a more standard solution ? Thanks Emmanuel -- Emmanuel Viennet: LIPN - Institut Galilee - Universite Paris-Nord 93430 Villetaneuse - France ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Thu Nov 21 06:32:15 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Thu, 21 Nov 1996 01:32:15 -0500 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays References: <199611181200.MAA11032@univ-paris13.fr> Message-ID: <199611210632.BAA08397@monty> (Crossposting from the matrix-sig to the Pythmain list.) > I am working with huge arrays (size comparable to the total swap > space of the machine, ie ~ 100Mo). > Currently, we write theses arrays to disk using the following > procedure: > fp.write( some_header_info ) > fp.write( arr.tostring() ) > or in some case: > fp.write( arr.byteswapped().tostring() ) > > This works but duplicates the array data to get a string. > > I know Python strings are immutables, so sharing data between a string > and an array is prohibited. > Should I write a C simple function > > write_array( arr, fp, byte_order ) > > directly writing the data to a file, swapping bytes if requested, > > or is there a more standard solution ? Hmm, interesting. Jack Jansen just mailed me an idea for a standard way to access "buffer-like" Python objects as linear sequences of bytes in Python, especially for I/O purposes. This should work for strings, arrays from the old array module, and numerical arrays, and other extension types that have an internal representation that is a contiguous sequence of bytes. (Jack's motivation was that it would make extensions for manipulating sound data easier to write; it seems the same problem as stated here.) Does the matrix community think this would be useful? Any thoughts on the form of the interface? I would presume the interface could be something like int PyBuffer_GetInfo(object *o, const char **address, int *length); to get the address and size of a "buffer" object into a C char pointer (or should that be void???) and length. The return value would be -1 if the object was not "bufferable". I guess a PyBuffer_Check(o) call should also be available. For *mutable* buffer objects, a version that returns a non-const pointer could be provided (but for strings this would fail) so one could read data *into* a buffer object. --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From johann@physics.berkeley.edu Thu Nov 21 07:59:04 1996 From: johann@physics.berkeley.edu (Johann Hibschman) Date: Wed, 20 Nov 1996 23:59:04 -0800 (PST) Subject: [PYTHON MATRIX-SIG] gist setup help? Message-ID: Hi all, I'm trying to get the gist module to work on my laptop running Linux, but it keeps dumping core on me. I can compile it and run the demo, but it crashes after displaying the first window with some sort of an X error. I have yorick working fine, and the rest of the Numeric module seems to work, but something's going terribly wrong. Any ideas? The only thing I can think of was that I changed the X directory path from /usr/X11R5 (or some such) to /usr/X11R6/lib in the lines for Setup from Setup.forgist. I'm running Python 1.4 with NumPy 1.0a5 (with Ellipses->Ellipsis), with the three patches from the gist distribution, and gistmodule 1.2. - Johann Johann Hibschman johann@physics.berkeley.edu ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From Emmanuel.Viennet@laforia.ibp.fr Thu Nov 21 11:00:17 1996 From: Emmanuel.Viennet@laforia.ibp.fr (VIENNET Emmanuel 48.06.27.97 Equipe Gallinari) Date: Thu, 21 Nov 1996 12:00:17 +0100 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: <199611210632.BAA08397@monty> References: <199611181200.MAA11032@univ-paris13.fr> <199611210632.BAA08397@monty> Message-ID: <199611211100.MAA04308@lpia2.ibp.fr> > I would presume the interface could be something like > > int PyBuffer_GetInfo(object *o, const char **address, int *length); > > to get the address and size of a "buffer" object into a C char pointer > (or should that be void???) and length. The return value would be -1 > if the object was not "bufferable". I guess a PyBuffer_Check(o) call > should also be available. For *mutable* buffer objects, a version > that returns a non-const pointer could be provided (but for strings > this would fail) so one could read data *into* a buffer object. Cool. This would allow very efficient implementation of matrix I/O packages. I presume PyBuffer_GetInfo() should fail on non-contiguous arrays ? PyBuffer_GetInfo() must check if the buffer is available read-only (immutable objects) or read/write (mutable, e.g. contiguous arrays). We could add it a mode argument, ala fopen() ('r', 'w'). Emmanuel -- Emmanuel Viennet: LIPN - Institut Galilee - Universite Paris-Nord 93430 Villetaneuse - France ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Thu Nov 21 12:04:04 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 21 Nov 96 13:04:04 +0100 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: <199611211100.MAA04308@lpia2.ibp.fr> (Emmanuel.Viennet@laforia.ibp.fr) Message-ID: <199611211203.NAA14724@ibs.ibs.fr> > I presume PyBuffer_GetInfo() should fail on non-contiguous arrays ? That would make it useless for many applications, since non-contiguous arrays can occur anywhere. > PyBuffer_GetInfo() must check if the buffer is available read-only > (immutable objects) or read/write (mutable, e.g. contiguous arrays). > > We could add it a mode argument, ala fopen() ('r', 'w'). How about a stream-like protocol, i.e. implementing read() and write()? That could be done for non-contiguous arrays, and still be efficient. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From Zachary_Roadhouse@brown.edu Thu Nov 21 13:48:18 1996 From: Zachary_Roadhouse@brown.edu (Zachary_Roadhouse@brown.edu) Date: Thu, 21 Nov 1996 08:48:18 -0500 (EST) Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: <199611211100.MAA04308@lpia2.ibp.fr> Message-ID: On 21-Nov-96 Emmanuel.Viennet@laforia.ibp.fr wrote: >Cool. This would allow very efficient implementation of matrix I/O >packages. > >I presume PyBuffer_GetInfo() should fail on non-contiguous arrays ? I agree with Konrad here. What would be needed for a non-continguous array would just be a little more book keeping. Read and write methods would be great! This feature would make it easier to pass data around to my labs non-Python crew. It would also might make it easier to transform a Numeric array into a PIL image (a common task in my group) by giving the Image a "file" to work with -- of course it'll be faster... Thumbs up here. - Zack E-MAIL: Zachary_Roadhouse@brown.edu WEB: http://althor.netspace.org/~zack/ Brown University, Box 220, Providence, RI 02912 Phone: (401) 863 - 5435 ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Thu Nov 21 16:26:45 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Thu, 21 Nov 1996 11:26:45 -0500 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: Your message of "Thu, 21 Nov 1996 13:04:04 +0100." <199611211203.NAA14724@ibs.ibs.fr> References: <199611211203.NAA14724@ibs.ibs.fr> Message-ID: <199611211626.LAA09612@monty> > How about a stream-like protocol, i.e. implementing read() and write()? > That could be done for non-contiguous arrays, and still be efficient. This makes some sense, except that it wouldn't (really) help for the situation where you have a HUGE array (HUGE == too big to copy in virtual memory) and you need to write it to disk. If the array is contiguous, a single write() system call could write the entire thing to disk. Next try (though this is getting complicated :-( ): how about a "sequence-of-buffers" interface. E.g.: long PyBuffer_Get(object *o, int i, const void **addres); would return the length of the i'th contiguous chunk, and store its address via *address; when there are no more chunks, it can return 0, and for non-buffering objects, it can return -1 and set an exception. --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Thu Nov 21 17:22:54 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 21 Nov 96 18:22:54 +0100 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: <199611211626.LAA09612@monty> (message from Guido van Rossum on Thu, 21 Nov 1996 11:26:45 -0500) Message-ID: <199611211722.SAA17160@ibs.ibs.fr> > > How about a stream-like protocol, i.e. implementing read() and write()? > > That could be done for non-contiguous arrays, and still be efficient. > > This makes some sense, except that it wouldn't (really) help for the > situation where you have a HUGE array (HUGE == too big to copy in > virtual memory) and you need to write it to disk. If the array is > contiguous, a single write() system call could write the entire thing > to disk. One solution would be to add some nice stream protocol to Python. Files would be one kind of stream, memory blocks another one. Then file.write(memory_block) should do the following: 1) The file object asks the memory block object for the next chunk of data (however large), imposing a maximum (which would actually be infinite in case of files). 2) The memory block object provides a pointer and length indicator for such a block and remembers where to continue, or an end-of-stream indication when there is no more data. 3) The file object writes the data to the disk file. 4) Goto 1. Such a protocol would transfer data in the largest blocks that both objects can handle. > Next try (though this is getting complicated :-( ): how about a > "sequence-of-buffers" interface. E.g.: I am not too fond of solutions that just try to work around one special problem. But of course it's better than raising exceptions for arrays. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Thu Nov 21 19:13:53 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Thu, 21 Nov 1996 14:13:53 -0500 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: Your message of "Thu, 21 Nov 1996 18:22:54 +0100." <199611211722.SAA17160@ibs.ibs.fr> References: <199611211722.SAA17160@ibs.ibs.fr> Message-ID: <199611211913.OAA09984@monty> > One solution would be to add some nice stream protocol to Python. > Files would be one kind of stream, memory blocks another one. > Then file.write(memory_block) should do the following: > > 1) The file object asks the memory block object for the next > chunk of data (however large), imposing a maximum (which > would actually be infinite in case of files). > > 2) The memory block object provides a pointer and length indicator for > such a block and remembers where to continue, or an end-of-stream > indication when there is no more data. > > 3) The file object writes the data to the disk file. > > 4) Goto 1. > > Such a protocol would transfer data in the largest blocks that both > objects can handle. > > > Next try (though this is getting complicated :-( ): how about a > > "sequence-of-buffers" interface. E.g.: > > I am not too fond of solutions that just try to work around one > special problem. But of course it's better than raising exceptions > for arrays. My proposal exactly matches your item (2), so we may be on to something... --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Fri Nov 22 10:01:15 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Fri, 22 Nov 96 11:01:15 +0100 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: <199611211913.OAA09984@monty> (message from Guido van Rossum on Thu, 21 Nov 1996 14:13:53 -0500) Message-ID: <199611221000.LAA19591@ibs.ibs.fr> > > > Next try (though this is getting complicated :-( ): how about a > > > "sequence-of-buffers" interface. E.g.: > > > > I am not too fond of solutions that just try to work around one > > special problem. But of course it's better than raising exceptions > > for arrays. > > My proposal exactly matches your item (2), so we may be on to > something... But the other items are also important. Suppose you want to transfer data between two streams, which each have restrictions on buffer size. Something must mediate between the two. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From guido@CNRI.Reston.Va.US Fri Nov 22 16:07:43 1996 From: guido@CNRI.Reston.Va.US (Guido van Rossum) Date: Fri, 22 Nov 1996 11:07:43 -0500 Subject: [PYTHON MATRIX-SIG] Re: Saving HUGE arrays In-Reply-To: Your message of "Fri, 22 Nov 1996 11:01:15 +0100." <199611221000.LAA19591@ibs.ibs.fr> References: <199611221000.LAA19591@ibs.ibs.fr> Message-ID: <199611221607.LAA11933@monty> > > My proposal exactly matches your item (2), so we may be on to > > something... > > But the other items are also important. Suppose you want to > transfer data between two streams, which each have restrictions > on buffer size. Something must mediate between the two. Hm. A proper stream protocol should never refuse to write more data than what fits in its buffer (it should just flush the buffer and refill it until it has processed all the data). When asked to read the next chunk it may return as much or as little as it has available in its buffer. But anyway, a stream protocol needs serious thought before it is added, because it has big ramifications. A buffer interface could be added with very little effort and could be immediately useful. --Guido van Rossum (home page: http://www.python.org/~guido/) ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From johann@physics.berkeley.edu Mon Nov 25 19:47:57 1996 From: johann@physics.berkeley.edu (Johann Hibschman) Date: Mon, 25 Nov 1996 11:47:57 -0800 (PST) Subject: [PYTHON MATRIX-SIG] Guidance for extensions? Message-ID: [ I sent this to the comp.lang.python as well, but I think you people are more likely to be able to answer my questions... ] Hello all, I've recently decided that Numeric Python all by itself is too slow for a project that I'm working on, so I'm trying to learn how to extend python. However, the docs I've found are rather confusing. I want to take a C++ program with classes like class Solver { public: int numPoints; Vector density, temperature; void DoTimeSteps( int numSteps ); }; and arrange a python module which would mirror the structure of the class. I want to be able to use python to set the variables, then call pass the arrays from python to the C++ object, call the C++ object to advance the solution by some number of timesteps, then pass the arrays back from the C++ object to the python object, so I can interactively play with them. This seems like it should be easy to do, but I'm not sure where to begin. The docs cover modules with simple functions fairly well, but I'm not sure how to construct C++ classes so they work, and I don't know how to transfer Numeric python arrays (or even normal python lists) to and from my program. Any suggestions, pointers, comments? - Johann Johann Hibschman johann@physics.berkeley.edu ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From janne@avocado.pc.helsinki.fi Tue Nov 26 16:25:23 1996 From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen) Date: 26 Nov 1996 18:25:23 +0200 Subject: [PYTHON MATRIX-SIG] A bug in 1.0a5 (ones, concatenate) Message-ID: This produces a segmentation fault: concatenate([ones([0,10]),ones([1,10])]) This does not: concatenate([zeros([0,10]),ones([1,10])]) I'm using Linux, Python 1.4 and 1.0a5 with fixed ellipses. -- Janne ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From hinsen@ibs.ibs.fr Tue Nov 26 18:14:10 1996 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 26 Nov 96 19:14:10 +0100 Subject: [PYTHON MATRIX-SIG] A bug in 1.0a5 (ones, concatenate) In-Reply-To: (message from Janne Sinkkonen on 26 Nov 1996 18:25:23 +0200) Message-ID: <199611261812.TAA09191@ibs.ibs.fr> > This produces a segmentation fault: > > concatenate([ones([0,10]),ones([1,10])]) > > This does not: > > concatenate([zeros([0,10]),ones([1,10])]) > > I'm using Linux, Python 1.4 and 1.0a5 with fixed ellipses. Same under HP/UX. And there are other operations with empty arrays that cause crashes. In case you wonder what the difference between ones() and zeros() is: zeros() is implemented in C; ones() is implemented in Python as, essentially, zero()+1. It might be the addition that ultimately causes the crash. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From barrett@compass.gsfc.nasa.gov Tue Nov 26 18:31:07 1996 From: barrett@compass.gsfc.nasa.gov (Paul Barrett) Date: Tue, 26 Nov 1996 13:31:07 -0500 Subject: [PYTHON MATRIX-SIG] NumPy benchmarks Message-ID: <199611261831.NAA25558@compass.gsfc.nasa.gov> Greetings, I've been doing some benchmarks of the fundamental array operations, e.g. '+', '*', 'cos', 'log', etc., and making comparisons to C code and IDL (RSI's Interactive Data Language). I thought it would be useful to have some numbers to show people. The results on a SPARCstation 20 with 64 MB RAM are in MFIPS (millions of floating-pointing instructions per second) where the array type is Float32 and lengths are in millions of elements. Hence, higher numbers are better. Addition and multiplication operations: Length C IDL Python 0.1 6.67 6.67 5.00 0.3 4.45 4.85 3.41 1.0 4.52 4.86 3.26 3.0 4.55 4.87 3.02 5.0 0.72 0.36 0.51 Cosine operation: Length C IDL Python 0.1 0.51 0.51 0.53 0.3 0.57 0.50 0.52 1.0 0.57 0.50 0.52 3.0 0.20 0.13 0.19 5.0 0.15 0.09 0.15 Tangent operation: Length C IDL Python 1.0 0.38 0.44 0.35 Logarithm operation: Length C IDL Python 1.0 0.75 0.70 0.65 1) As the numbers indicate, the binary operations show room for improvement. They are about 33% slower than the C and IDL code. The unary operations are typically within 10% and therefore are satisfactory. Does anyone have any suggestions about how the binary ops can be improved? 2) Also note that the numbers become worse as the arrays become a sizeable fraction of the computer's RAM size. The addition of two arrays in a third array of 5 million elements each requires 60 MB of out of 64 MB of RAM. Thus the results become dependant on the time needed to swap memory. At present, it's not clear to me if improvements can be made in this area, but it would be a nice feather in Python's cap if improvements could be made for very large arrays. 3) Having had sometime to work with NumPy recently, I have three comments about the implementation: The first is the use of 'arccos', 'arcsin', etc. instead of using 'acos', 'asin', etc. This seems to me to be both inconsistent with Python's math module which uses 'acos', 'asin', etc. and with common practice. Is there a reason for this? The second is about the 'arrayrange' method. It is very slow for very large arrays. During my initial tests, I found Python to be very slow compared to C and IDL. I then discovered the reason to be my use of arrayrange to create arrays. As the source code notes, this method should be converted to C, since it can be used heavily in some numerical algorithms. And the third is the difference between umathmodule.c and fast_umathmodule.c. Currently there is none. Will someone please outline what they should be? Cheers, Paul |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| | Paul Barrett - Astrophysicist - Universities Space Research Association | | Compton Observatory Science Support Center | | NASA/Goddard SFC phone: 301-286-1108 "Cracking toast, Gromit" | | Code 660.1, FAX: 301-286-1681 says Wallace. | | Greenbelt,MD 20771 barrett@compass.gsfc.nasa.gov (in The Wrong Trousers) | | http://lheawww.gsfc.nasa.gov/users/barrett/CV.html | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From johann@physics.berkeley.edu Tue Nov 26 19:05:53 1996 From: johann@physics.berkeley.edu (Johann Hibschman) Date: Tue, 26 Nov 1996 11:05:53 -0800 (PST) Subject: [PYTHON MATRIX-SIG] SWIG, shadow classes, and NumPy Message-ID: Hi all, In my continuing struggles to get my hydro code working and working fast, I have lapsed into straight C++ with a Python shadow class generated by SWIG. So far that works beautifully; I can get at all of my data and I can run the computational methods of my class. The one problem is that all of the vectors come out as specialized objects for which I have to define the __getitem__ and __setitem__ methods explicitly in the SWIG .i file. It would be nice to be able to use all of the interface options available to standard NumPy arrays (math operations, assignment by a scalar, slices, etc.) and have these operations afffect the actual data in my C++ class. One way to do this would be to mirror all of the data arrays as NumPy arrays in the shadow class, but this leads to problems with the shadow class getting out of synch with the C++ class. I'm not sure exactly how I'd implement this, but I'm confident that I could after reading through the NumPy source code a few times. (Not that I really want to do that.) Is there another, more clever way which someone more familiar with the ins and outs of the NumPy innards can see to do this? Could I use a NumPy array in the native C++ code? Could I create a class wrapper for a NumPy array so it would be usable by C++? Could I delude a NumPy array object into thinking that the block of data in the C++ vector belonged to it instead, without having the NumPy array attempt to delete the data at some point? Could I introduce SWIG directives that would somehow map from one to the other at runtime? Thanks, - Johann Johann Hibschman johann@physics.berkeley.edu ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From tbyang@icf.llnl.gov Tue Nov 26 20:37:00 1996 From: tbyang@icf.llnl.gov (Tser-Yuan (Brian) Yang) Date: Tue, 26 Nov 1996 12:37:00 -0800 (PST) Subject: [PYTHON MATRIX-SIG] Re: SWIG, shadow classes, and NumPy Message-ID: Since SWIG shadows a pointer with a Python string object, I think the following can be useful: In your C++ extension, implement a method 'Ptr_to_PyArray' that takes three arguments, i.e., a pointer object for the data, a pointer object for the shape, and a Python integer object for the number of dimension, and returns a PyArray object of the Numeric module. This can be done with the API 'PyArray_FromDimsAndData' in 'arrayobject.c'. An inverse method 'PyArray_to_Ptr' that takes a PyArray object and returns a tuple may be very useful, too. I have been thinking about implementing these interface functions in one of our applications, but I am putting it off because my current project don't need them. Brian Yang ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org ================= From chris.chase@jhuapl.edu Fri Nov 29 20:38:39 1996 From: chris.chase@jhuapl.edu (Chris Chase SRM) Date: Fri, 29 Nov 1996 15:38:39 -0500 Subject: [PYTHON MATRIX-SIG] changes to fix __array__() Message-ID: <199611292041.PAA25686@python.org> I am using NumPy-1.0a5.tar and Python 1.4. I was trying to use the __array__(self, typecode=None) interface for getting an object as an array and found that it did not work. I traced the problem to array_fromobject() in arrayobject.c. I have made the necessary changes to array_fromobject() in arrayobject.c. I also made corrections to refecrence handling of Python objects in the same function (according to Lutz's PP book both PyEval_CallObject() and PyObject_GetAttrString() pass pass owenership of result, i.e. it is already INCREF'd and must be DECREF'd when finished). As a result of all this I found that asarray() did not use PyArray_FromObject() but instead copies via PyArray_CopyFromObject when using the __array__() interface. To make it use PyArray_FromObject(), I moved asarray() in multiarray.c instead of in Numeric.py. An example of the fixed behavior of the __array__ interface and asarray(): class a: def __init__(self): self.a = arange(8) def __array__(self,t=None): if t: return asarray(self.a,t) return asarray(self.a) b = a() c = array(b) c[2] = -9 >>> c 0 1 -9 3 4 5 6 7 >>> b.a 0 1 2 3 4 5 6 7 d = asarray(b) d[2] = -9 >>> b.a 0 1 -9 3 4 5 6 7 According to the documentation for array the interface is given as "array(sequence, typecode=None)" which seems to indicate that None is valid value for typecode. However, None is not a valid value and typecode is not a keyword argument. The documentation should be changed. Although, I think that it would be better to let typecode be a possible keyword argument and let None be a valid value that gets mapped to PyArray_NOTYPE (in arrayobject.h). Anyway, for the __array__ and asarray changes I modified arrayobject.c, multiarray.c and Numeric.py. The context diffs created in the NumPy directory are below, which I believe are suitable for the patch program, are below. Chris Chase =================================================================== RCS file: arrayobject.c,v retrieving revision 1.1 diff -c5 -r1.1 arrayobject.c *** arrayobject.c 1996/11/28 07:10:25 1.1 --- arrayobject.c 1996/11/29 19:13:11 *************** *** 1612,1641 **** } #define ByCopy 1 #define Contiguous 2 ! PyObject *array_fromobject(PyObject *op, int type, int min_depth, int max_depth, int flags) { ! PyObject *r; ! r = NULL; ! if (!PyArray_Check(op) && PyObject_HasAttrString(op, "__array__")) { PyObject *ap, *arglist; if (type == PyArray_NOTYPE) { arglist = Py_BuildValue("()"); } else { arglist = Py_BuildValue("(c)", type); } ! ap = PyObject_GetAttrString(op, "__array__"); ! r = PyEval_CallObject(ap, arglist); Py_DECREF(arglist); ! ! Py_XINCREF(r); ! } ! if (type == PyArray_NOTYPE) { type = PyArray_ObjectType(op, 0); } if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT || --- 1612,1646 ---- } #define ByCopy 1 #define Contiguous 2 ! PyObject *array_fromobject(PyObject *op_in, int type, int min_depth, int max_depth, int flags) { ! PyObject *r, *op; ! r = NULL; ! if (!PyArray_Check(op_in) && PyObject_HasAttrString(op_in, "__array__")) { ! /* __array__(self, type=None) method interface ! for getting an object as an array. */ PyObject *ap, *arglist; if (type == PyArray_NOTYPE) { arglist = Py_BuildValue("()"); } else { arglist = Py_BuildValue("(c)", type); } ! ap = PyObject_GetAttrString(op_in, "__array__"); ! op = PyEval_CallObject(ap, arglist); ! Py_DECREF(ap); Py_DECREF(arglist); ! if (op == NULL) return NULL; ! } else { ! op = op_in; ! Py_INCREF(op); ! } ! if (type == PyArray_NOTYPE) { type = PyArray_ObjectType(op, 0); } if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT || *************** *** 1648,1673 **** Py_INCREF(op); r = op; } } else { if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num; ! if ((r = PyArray_Cast((PyArrayObject *)op, type)) == NULL) return NULL; } } else { r = Array_FromSequence(op, type, min_depth,max_depth); ! if (r == NULL) { ! if (min_depth <= 0) { ! PyErr_Clear(); ! r = PyArray_FromScalar(op, type); ! } else { ! return NULL; ! } } } if (r == NULL) return NULL; ! if(!PyArray_Check(r)) { PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array"); return NULL; } if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) { Py_DECREF(r); --- 1653,1677 ---- Py_INCREF(op); r = op; } } else { if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num; ! r = PyArray_Cast((PyArrayObject *)op, type); } } else { r = Array_FromSequence(op, type, min_depth,max_depth); ! if (r == NULL && min_depth <= 0) { ! PyErr_Clear(); ! r = PyArray_FromScalar(op, type); } } + /* finished with op */ + Py_DECREF(op); + if (r == NULL) return NULL; ! if (!PyArray_Check(r)) { PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array"); return NULL; } if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) { Py_DECREF(r); rcsdiff: RCS/multiarray.c,v: No such file or directory rcsdiff: Numeric/RCS/Numeric.py,v: No such file or directory custer [121]> ../misc/makepatch ../misc/makepatch =================================================================== RCS file: arrayobject.c,v retrieving revision 1.1 diff -c5 -r1.1 arrayobject.c *** arrayobject.c 1996/11/28 07:10:25 1.1 --- arrayobject.c 1996/11/29 19:13:11 *************** *** 1612,1641 **** } #define ByCopy 1 #define Contiguous 2 ! PyObject *array_fromobject(PyObject *op, int type, int min_depth, int max_depth, int flags) { ! PyObject *r; ! r = NULL; ! if (!PyArray_Check(op) && PyObject_HasAttrString(op, "__array__")) { PyObject *ap, *arglist; if (type == PyArray_NOTYPE) { arglist = Py_BuildValue("()"); } else { arglist = Py_BuildValue("(c)", type); } ! ap = PyObject_GetAttrString(op, "__array__"); ! r = PyEval_CallObject(ap, arglist); Py_DECREF(arglist); ! ! Py_XINCREF(r); ! } ! if (type == PyArray_NOTYPE) { type = PyArray_ObjectType(op, 0); } if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT || --- 1612,1646 ---- } #define ByCopy 1 #define Contiguous 2 ! PyObject *array_fromobject(PyObject *op_in, int type, int min_depth, int max_depth, int flags) { ! PyObject *r, *op; ! r = NULL; ! if (!PyArray_Check(op_in) && PyObject_HasAttrString(op_in, "__array__")) { ! /* __array__(self, type=None) method interface ! for getting an object as an array. */ PyObject *ap, *arglist; if (type == PyArray_NOTYPE) { arglist = Py_BuildValue("()"); } else { arglist = Py_BuildValue("(c)", type); } ! ap = PyObject_GetAttrString(op_in, "__array__"); ! op = PyEval_CallObject(ap, arglist); ! Py_DECREF(ap); Py_DECREF(arglist); ! if (op == NULL) return NULL; ! } else { ! op = op_in; ! Py_INCREF(op); ! } ! if (type == PyArray_NOTYPE) { type = PyArray_ObjectType(op, 0); } if (PyArray_Check(op) && (((PyArrayObject *)op)->descr->type_num != PyArray_OBJECT || *************** *** 1648,1673 **** Py_INCREF(op); r = op; } } else { if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num; ! if ((r = PyArray_Cast((PyArrayObject *)op, type)) == NULL) return NULL; } } else { r = Array_FromSequence(op, type, min_depth,max_depth); ! if (r == NULL) { ! if (min_depth <= 0) { ! PyErr_Clear(); ! r = PyArray_FromScalar(op, type); ! } else { ! return NULL; ! } } } if (r == NULL) return NULL; ! if(!PyArray_Check(r)) { PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array"); return NULL; } if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) { Py_DECREF(r); --- 1653,1677 ---- Py_INCREF(op); r = op; } } else { if (type > PyArray_NTYPES) type = PyArray_DescrFromType(type)->type_num; ! r = PyArray_Cast((PyArrayObject *)op, type); } } else { r = Array_FromSequence(op, type, min_depth,max_depth); ! if (r == NULL && min_depth <= 0) { ! PyErr_Clear(); ! r = PyArray_FromScalar(op, type); } } + /* finished with op */ + Py_DECREF(op); + if (r == NULL) return NULL; ! if (!PyArray_Check(r)) { PyErr_SetString(PyExc_ValueError, "Internal error array_fromobject not producing an array"); return NULL; } if (min_depth != 0 && ((PyArrayObject *)r)->nd < min_depth) { Py_DECREF(r); rcsdiff: RCS/multiarray.c,v: No such file or directory =================================================================== RCS file: Numeric/Numeric.py,v retrieving revision 1.1 diff -c5 -r1.1 Numeric/Numeric.py *** Numeric/Numeric.py 1996/11/29 20:24:40 1.1 --- Numeric/Numeric.py 1996/11/29 20:25:12 *************** *** 31,40 **** --- 31,41 ---- return m.astype(typecode) else: return m #Include some functions straight from multiarray + asarray = multiarray.asarray array = multiarray.array zeros = multiarray.zeros fromstring = multiarray.fromstring take = multiarray.take reshape = multiarray.reshape *************** *** 235,251 **** def nonzero(a): """Return the indices of the elements of a which are not zero, a must be 1d """ return repeat(arange(len(a)), not_equal(a, 0)) ! def asarray(a, typecode=None): ! if type(a) == arraytype and (typecode == None or typecode == a.typecode()): ! return a ! elif typecode == None: ! return array(a) ! else: ! return array(a, typecode) #Move this into C to do it right! def shape(a): return asarray(a).shape --- 236,252 ---- def nonzero(a): """Return the indices of the elements of a which are not zero, a must be 1d """ return repeat(arange(len(a)), not_equal(a, 0)) ! ## def asarray(a, typecode=None): ! ## if type(a) == arraytype and (typecode == None or typecode == a.typecode()): ! ## return a ! ## elif typecode == None: ! ## return array(a) ! ## else: ! ## return array(a, typecode) #Move this into C to do it right! def shape(a): return asarray(a).shape ================= MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org =================