From hinsen@cnrs-orleans.fr Mon Mar 1 11:05:30 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 1 Mar 1999 12:05:30 +0100 Subject: [Matrix-SIG] Tutorial on using NumPy arrays in C extension modules Message-ID: <199903011105.MAA14708@dirac.cnrs-orleans.fr> The question of how to use NumPy arrays in C extensions comes up regularly on comp.lang.python and on the Matrix-SIG, because there is almost no documentation other than the source code. On the occasion of an advanced Python workshop at the EMBL in Heidelberg, I wrote a tutorial on this subject, which is now available in my Starship cabin: http://starship.python.net/crew/hinsen/NumPyExtensions.html -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Oliphant.Travis@mayo.edu Tue Mar 2 00:14:39 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 1 Mar 1999 18:14:39 -0600 (EST) Subject: [Matrix-SIG] Documentation for Numerical Python Extension writing Message-ID: Well, it looks like two of us have been working on explaining how to write the extensions in C using the arrayobject specific calls. My beginning attempt at documentation for the C-API of the arrayobject and the ufuncobject can be found in postscript format at the link below. So far all of the C-API arrayobject calls are explained (as far as I can with what I have learned). I still need to add the Ufunc API calls and explain the ufunc structure. Here is the link: http://oliphant.netpedia.net/packages/Numerical_Extensions.ps.gz I would appreciate feedback if you are interested in taking a look. My approach is a little different than Konrad's in that it is less tutorial in nature as it trys to document all of the calls. It exposes all of the API even though a typical user would not use all of the calls. It's good to see some work being done on documenting the NumPy internals. Thanks Konrad, -Travis From Oliphant.Travis@mayo.edu Tue Mar 2 00:28:26 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 1 Mar 1999 18:28:26 -0600 (EST) Subject: [Matrix-SIG] PyArray_Free in arrayobject.c Message-ID: While I was looking at the source to write some documentation, I noticed that PyArray_Free looks like it won't work as intended. It appears the PyArray_Free is supposed to work along with PyArray_As2D and PyArray_As1D to dereference objects and free up memory that those two functions create. It probably works as intended for 1D arrays since it is called often after use of PyArray_As1D throughout multiarraymodule.c and arrayobject.c But, it doesn't look like it would work for 2D arrays since the following code which deallocates the array of pointers created by PyArray_As2D never gets called. extern int PyArray_Free(PyObject *op, char *ptr) { PyArrayObject *ap = (PyArrayObject *)op; int i, n; if (ap->nd > 2) return -1; if (ap->nd == 3) { /* This statement won't be true if ap->nd <= 2 which it must be in order to reach this point in the code. So the following code never gets called??*/ n = ap->dimensions[0]; for (i=0; ind >= 2) { free(ptr); } Py_DECREF(ap); return 0; } Has anyone successfully used PyArray_Free along with PyArray_As2D. For now it looks like using PyArray_As2D springs a memory leak. Anyone know what's going on here. -Travis From da@ski.org Tue Mar 2 00:41:58 1999 From: da@ski.org (David Ascher) Date: Mon, 1 Mar 1999 16:41:58 -0800 (Pacific Standard Time) Subject: [Matrix-SIG] PyArray_Free in arrayobject.c In-Reply-To: Message-ID: On Mon, 1 Mar 1999, Travis Oliphant wrote: > While I was looking at the source to write some documentation, I noticed > that PyArray_Free looks like it won't work as intended. I asked Jim Hugunin about it a few months back and submitted a (flawed) patch suggestion. His answer: >> 2) Here is PyArray_Free: > This code admittedly looks a little bit strange. I believe that it > should only be called very rarely (at least as originally inteneded) > and is supposed to be partnered with the PyArray_As1d and As_2d (or > something like that) calls -- the reason for the skipped nd == 3 case > is that I had originally planned an As3d function and never got around > to implementing it. > The As2d function in particular created a new C-array which isn't > referenced by any PyArray objects (so people can do a[0][0] C-style > indexing). The Free function handles freeing this. Your new version > wouldn't do the right thing with 1d arrays (the data area is not > supposed to be freed by this function). > I can't imagine why PyArray_Reshape, PyArray_Transpose, > PyArray_Repeat, ... would be using this function, but if they are it > might indicate something peculiar is going on. > BTW - A huge motivating factor of JPython was how frustrated I got > with all these silly refcount issues when building numpy. I wouldn't > be surprised if you found some mistakes on that score... --david From godzilla@netmeg.net (Les Schaffer) Tue Mar 2 01:19:19 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Mon, 1 Mar 1999 20:19:19 -0500 (EST) Subject: [Matrix-SIG] Documentation for Numerical Python Extension writing In-Reply-To: References: Message-ID: <14043.15511.679728.247391@netmeg.net> Travis says: > I would appreciate feedback if you are interested in taking a look. quick glance: definitely a start in the right direction. what i would really like to see, especially with your aproach as opposed to Konrad's slides, is to have (some of) this stuff go in the darn SOURCE CODE!!! NumPy is all about homogeneous arrays of numbers (etc). its about an array object. we learn about array objects in arrayobject.h. lets go see whats in arrayobject.h about array objects: typedef struct { PyObject_HEAD char *data; int nd; int *dimensions, *strides; PyObject *base; PyArray_Descr *descr; int flags; } PyArrayObject; notice, not a comment in sight. even something as stupid as ... char * data; /* here's where the hallowed data lie. */ /* if you want access as array[i][j] see */ /* PyArray_As2D below */ ... can lend sustenance to a poor wayfarer looking through the code for the first time in preparation for doing a SWIG job to wrap array objects from python into their 3D weather forecasting code . we can do better NumPy docs, as evidenced by travis' and konrad's stuff. but doesnt some of this documentation belong IN THE SOURCE CODE????? is there some way to accrete the acumulated wisdom of NumPy users directly into the source, where many people who do numerical work will ultimately look for guidance, consolation, spiritual values, and entertainment. or how about an annotated HTML version of the numpy code, with comments, URL links to other sites with examples of use of particular pieces of the code (how DO I USE PyArray_As2D??????, oh here's a way ... arrayobject.h: ================ ... extern PyArray_Take_RET PyArray_Take PyArray_Take_PROTO; extern PyArray_As1D_RET PyArray_As1D PyArray_As1D_PROTO; /* this function lets you access 2D arrays in their rightful manner: array[i][j]. URL: http://www.docs.org/NumPy/PyArray_As2D/you_wont_believe_how_easy_it_is.html */ extern PyArray_As2D_RET PyArray_As2D PyArray_As2D_PROTO; .... i hereby offer time as in a few hours per week to help do something like this, but only if its somehow officially tied in with NumPy so the docs are current, accurate, timely, etc, and, um, uh, official. nothing worse than a documenttaion branch and a source code branch. les schaffer -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From Oliphant.Travis@mayo.edu Tue Mar 2 05:01:17 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 1 Mar 1999 23:01:17 -0600 (EST) Subject: [Matrix-SIG] PyArray_Free in arrayobject.c In-Reply-To: Message-ID: On Mon, 1 Mar 1999, David Ascher wrote: > > > This code admittedly looks a little bit strange. I believe that it > > should only be called very rarely (at least as originally inteneded) > > and is supposed to be partnered with the PyArray_As1d and As_2d (or > > something like that) calls -- the reason for the skipped nd == 3 case > > is that I had originally planned an As3d function and never got around > > to implementing it. I suspected that PyArray_Free was supposed to be partnered with PyArray_AsXd. I see why there would be an nd == 3 case that would get skipped. Still, the code there now seems to free a single array of pointers (exactly what it should do for the n=2 case) > > The As2d function in particular created a new C-array which isn't > > referenced by any PyArray objects (so people can do a[0][0] C-style > > indexing). The Free function handles freeing this. This is what I thought it was supposed to do. But, as is, this array of pointers never gets released because ap->nd != 3, and the code to release this array of pointers is inside of a conditional branch that requires ap->nd == 3. > I can't imagine why PyArray_Reshape, PyArray_Transpose, > PyArray_Repeat, ... would be using this function, but if they are it > might indicate something peculiar is going on. Well, they are indeed being used in these functions. I agree that they are unnecessary though. Is something peculiar going on? > BTW - A huge motivating factor of JPython was how frustrated I got > with all these silly refcount issues when building numpy. I wouldn't > be surprised if you found some mistakes on that score... Is there a NumPy for JPython? -- Travis From da@ski.org Tue Mar 2 05:09:28 1999 From: da@ski.org (David Ascher) Date: Mon, 1 Mar 1999 21:09:28 -0800 (Pacific Standard Time) Subject: [Matrix-SIG] PyArray_Free in arrayobject.c In-Reply-To: Message-ID: On Mon, 1 Mar 1999, Travis Oliphant wrote: > This is what I thought it was supposed to do. But, as is, this array of > pointers never gets released because ap->nd != 3, and the code to release > this array of pointers is inside of a conditional branch that requires > ap->nd == 3. I'm willing to believe there's a memory leak. I just haven't analyzed the code deeply enough or tested it. > > BTW - A huge motivating factor of JPython was how frustrated I got > > with all these silly refcount issues when building numpy. I wouldn't > > be surprised if you found some mistakes on that score... > > Is there a NumPy for JPython? Sort of -- Tim Hochberg has started a reimplementation in JPython. I haven't heard much about it in a while though. See: http://www.egroups.com/list/jnumeric --david From hinsen@cnrs-orleans.fr Tue Mar 2 14:10:36 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Tue, 2 Mar 1999 15:10:36 +0100 Subject: [Matrix-SIG] Documentation for Numerical Python Extension writing In-Reply-To: (message from Travis Oliphant on Mon, 1 Mar 1999 18:14:39 -0600 (EST)) References: Message-ID: <199903021410.PAA02272@dirac.cnrs-orleans.fr> > Well, it looks like two of us have been working on explaining how to write > the extensions in C using the arrayobject specific calls. Better two than nobody! > Here is the link: > http://oliphant.netpedia.net/packages/Numerical_Extensions.ps.gz Looks... ugly. Hardly readable in fact. Do I need some Lyx-specific TeX macro packages to get a decent result? For the moment the best way to read it is reading the LaTeX source directly. The contents, however, are much better! And I hope that you will continue with the ufunc stuff, about which I know nothing, but which could be very useful for some projects. It's also worthwhile to have a decent description of how to use SWIG typemaps. I have never used this either, but I could imagine that in many cases it is the simplest solution. > I would appreciate feedback if you are interested in taking a look. My > approach is a little different than Konrad's in that it is less tutorial > in nature as it trys to document all of the calls. It exposes all of the > API even though a typical user would not use all of the calls. Which raises an important question. I did not attempt to document the full C API for three reasons: 1) laziness (the main one) 2) in order not to intimidate new users 3) because I don't know how much is considered "official", in the sense of supported in future releases. Of course there should be a reference guide in addition to a tutorial, but the question of what will be the "official" C API remains to be answered. Some functions in NumPy seem of little use to end users, so it's better not to consider them official, to keep more freedom for future improvements. Other functions are hard to understand and/or buggy; the borderline is unclear. Perhaps the best approach is to divide a reference manual into "official" and "known to work, but perhaps not forever" sections. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Ralph.Heinkel@EMBL-Heidelberg.DE Tue Mar 2 14:40:08 1999 From: Ralph.Heinkel@EMBL-Heidelberg.DE (Ralph.Heinkel@EMBL-Heidelberg.DE) Date: Tue, 2 Mar 1999 15:40:08 +0100 Subject: [Matrix-SIG] Documentation for Numerical Python Extension writing In-Reply-To: <199903021410.PAA02272@dirac.cnrs-orleans.fr> References: <199903021410.PAA02272@dirac.cnrs-orleans.fr> Message-ID: <199903021440.PAA23192@hobbes.embl-heidelberg.de> On Tue 2 Mar 99, Konrad Hinsen writes: > > Well, it looks like two of us have been working on explaining how to write > > the extensions in C using the arrayobject specific calls. > > Better two than nobody! True! > > > Here is the link: > > http://oliphant.netpedia.net/packages/Numerical_Extensions.ps.gz > > Looks... ugly. Hardly readable in fact. Do I need some Lyx-specific > TeX macro packages to get a decent result? For the moment the best > way to read it is reading the LaTeX source directly. > Maybe you just need a working postscript viewer; printing the document or looking at it with ghostview is just fine. Ralph -- ------------------------------------------------------ Ralph Heinkel European Molecular Biology Laboratory (EMBL) Meyerhofstr. 1 69012 Heidelberg Tel. +49 6221/387 529 eMail: heinkel@embl-heidelberg.de From phil@geog.ubc.ca Thu Mar 4 00:20:49 1999 From: phil@geog.ubc.ca (Phil Austin) Date: Wed, 3 Mar 1999 16:20:49 -0800 Subject: [Matrix-SIG] Documentation for Numerical Python Extension writing In-Reply-To: <199903021410.PAA02272@dirac.cnrs-orleans.fr> References: <199903021410.PAA02272@dirac.cnrs-orleans.fr> Message-ID: <199903040020.QAA14624@brant.geog.ubc.ca> Konrad Hinsen writes: > > Looks... ugly. Hardly readable in fact. Do I need some Lyx-specific > TeX macro packages to get a decent result? For the moment the best > way to read it is reading the LaTeX source directly. > I took Travis' TeX file, re-ran it using dvips with the free CM scalable fonts, then put it through Adobe distiller. It's available at: ftp://ftp.geog.ubc.ca/pub/python/Numerical_Extensions.pdf Phil From Oliphant.Travis@mayo.edu Thu Mar 4 10:16:40 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Thu, 4 Mar 1999 04:16:40 -0600 (EST) Subject: [Matrix-SIG] Numerical Extensions in C Documentation Updated Message-ID: Hello, I just updated some unofficial reference documentation I've been working on for the C API to Numerical Python. It is now available as a PDF file or in LATEX source format (both gzipped) at. http://oliphant.netpedia.net/packages/Numerical_Extensions.pdf.gz http://oliphant.netpedia.net/packages/Numerical_Extensions.tex.gz I haven't changed the arrayobject portions of the documentation much. I just added some detailed information about the ufuncobject interface. It is fairly complete but unpolished. Have a look if you are interested in writing C extensions modules. Please send any feedback you have. --Travis From thomas.hauser@usa.net Fri Mar 5 20:01:27 1999 From: thomas.hauser@usa.net (Thomas Hauser) Date: Fri, 05 Mar 1999 15:01:27 -0500 Subject: [Matrix-SIG] Problems using PyPDB with Python1.5.1 and 1.5.2b2 Message-ID: <36E03816.AFD0ECD5@usa.net> Hi I tried using PyPDB from the Numeric distribution. this is the output I get if I try running Demo/PDtest.py Contents of test.pdb: ['a', 'ax', 'axx', 'c', 'cx', 'cxx', 'd', 'dx', 'dxx', 'k', 'kx', 'kxx', 'x', 'y'] f.inquire_ls('d*') = ('d', 'dx', 'dxx') f.inquire_ls('','complex') = ('c', 'cx', 'cxx') Variables: f.x = Segmentation fault (core dumped) The version is LLNLDistribution9 and it didn't work with python 1.5.1 and 1.5.2b2 on a sgi IRIX6.4. Any hints? Thomas From dubois1@llnl.gov Fri Mar 5 22:45:29 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 5 Mar 1999 14:45:29 -0800 Subject: [Matrix-SIG] Problems using PyPDB with Python1.5.1 and 1.5.2b2 Message-ID: <000b01be6759$dea5a320$f4160218@c1004579-c.plstn1.sfba.home.com> Hi, I just took a quick look and it appears I didn't yet go back and added the import_array() statement to the init routine. I'll fix it in the next release. Paul -----Original Message----- From: Thomas Hauser To: matrix-sig@python.org Date: Friday, March 05, 1999 11:58 AM Subject: [Matrix-SIG] Problems using PyPDB with Python1.5.1 and 1.5.2b2 >Hi > >I tried using PyPDB from the Numeric distribution. > >this is the output I get if I try running Demo/PDtest.py > >Contents of test.pdb: >['a', > 'ax', > 'axx', > 'c', > 'cx', > 'cxx', > 'd', > 'dx', > 'dxx', > 'k', > 'kx', > 'kxx', > 'x', > 'y'] >f.inquire_ls('d*') = >('d', 'dx', 'dxx') >f.inquire_ls('','complex') = >('c', 'cx', 'cxx') >Variables: >f.x = >Segmentation fault (core dumped) > >The version is LLNLDistribution9 and it didn't work with python 1.5.1 >and 1.5.2b2 on a sgi IRIX6.4. Any hints? > >Thomas > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From Les Schaffer Fri Mar 5 23:31:39 1999 From: Les Schaffer (Les Schaffer) Date: Fri, 5 Mar 1999 18:31:39 -0500 (EST) Subject: [Matrix-SIG] Re: Problems using PyPDB with Python1.5.1 and 1.5.2b2 In-Reply-To: <000b01be6759$dea5a320$f4160218@c1004579-c.plstn1.sfba.home.com> References: <000b01be6759$dea5a320$f4160218@c1004579-c.plstn1.sfba.home.com> Message-ID: <14048.26971.620368.677661@netmeg.net> PD wrote: > I just took a quick look and it appears I didn't yet go back and > added the import_array() statement to the init routine. I'll fix it > in the next release. dont this do the trick (it does for me): (gustav)/usr/local/src/LLNLDistribution/PyPDB/: diff -c Src/pypdbmodule.c.orig Src/pypdbmodule.c *** Src/pypdbmodule.c.orig Fri Mar 5 18:24:33 1999 --- Src/pypdbmodule.c Fri Mar 5 18:29:49 1999 *************** *** 1301,1306 **** --- 1301,1308 ---- pypdb_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); + import_array(); + /* Add some symbolic constants to the module */ d = PyModule_GetDict(m); From dubois1@llnl.gov Sat Mar 6 03:09:26 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 5 Mar 1999 19:09:26 -0800 Subject: [Matrix-SIG] Re: Problems using PyPDB with Python1.5.1 and 1.5.2b2 Message-ID: <001801be677e$be4cd880$f4160218@c1004579-c.plstn1.sfba.home.com> Yes, that's what I meant. The exact placement of the statement isn't important, as long as the init calls it. -----Original Message----- From: Les Schaffer To: matrix-sig@python.org Date: Friday, March 05, 1999 3:31 PM Subject: [Matrix-SIG] Re: Problems using PyPDB with Python1.5.1 and 1.5.2b2 >PD wrote: > >> I just took a quick look and it appears I didn't yet go back and >> added the import_array() statement to the init routine. I'll fix it >> in the next release. > >dont this do the trick (it does for me): > > >(gustav)/usr/local/src/LLNLDistribution/PyPDB/: diff -c Src/pypdbmodule.c.orig Src/pypdbmodule.c >*** Src/pypdbmodule.c.orig Fri Mar 5 18:24:33 1999 >--- Src/pypdbmodule.c Fri Mar 5 18:29:49 1999 >*************** >*** 1301,1306 **** >--- 1301,1308 ---- > pypdb_module_documentation, > (PyObject*)NULL,PYTHON_API_VERSION); > >+ import_array(); >+ > /* Add some symbolic constants to the module */ > d = PyModule_GetDict(m); > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From dars@fook.mechanoid.soton.ac.uk Mon Mar 8 15:11:19 1999 From: dars@fook.mechanoid.soton.ac.uk (Dave Stinchcombe) Date: Mon, 8 Mar 1999 15:11:19 +0000 Subject: [Matrix-SIG] returning arrays Message-ID: <19990308151119.I14044@fook.mechanoid.soton.ac.uk> Hi Folks, it's that time in my work when graphs and charts need to be drawn so I'm palying with NumPy again. There is one thing I would like to do, but can't figure out. I would like to write a function which accepts a Numeric.array as it's argument and returns a Numeric.array, in exactly the kind of way that Numeric.cos does. Can any one tell me the trick. My function is just a shortish bit of numerics that requires only one parameter in order to get a solution. Yours Dave From dubois1@llnl.gov Mon Mar 8 15:53:32 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Mon, 8 Mar 1999 07:53:32 -0800 Subject: [Matrix-SIG] returning arrays Message-ID: <001c01be697b$d1435fa0$f4160218@c1004579-c.plstn1.sfba.home.com> Forgive me if any of this is misspelled, but it is approximately right: To create and return a 1-D array of length n: PyArray_TYPES t; PyArray* p; int n; double* data; n = ....; t = PyArray_DOUBLE; p = PyArray_FromDims(1, &n, t); data = p->data; ...data[i] = ...; return (PyObject *) p; -----Original Message----- From: Dave Stinchcombe To: matrix-sig@python.org Date: Monday, March 08, 1999 7:07 AM Subject: [Matrix-SIG] returning arrays >Hi Folks, >it's that time in my work when graphs and charts need to be drawn so I'm >palying with NumPy again. There is one thing I would like to do, but can't >figure out. I would like to write a function which accepts a Numeric.array >as it's argument and returns a Numeric.array, in exactly the kind of way >that Numeric.cos does. Can any one tell me the trick. My function is just a >shortish bit of numerics that requires only one parameter in order to get a >solution. > >Yours >Dave > > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From dubois1@llnl.gov Mon Mar 8 16:04:14 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Mon, 8 Mar 1999 08:04:14 -0800 Subject: [Matrix-SIG] returning arrays part 2 Message-ID: <001f01be697d$504cee00$f4160218@c1004579-c.plstn1.sfba.home.com> For some reason I only answered half your question. As regards the input: You do a normal PyTuple_Parse with code "O". That gives you a PyObject *pobj; Then: PyArray *pin; double* din; pin = PyArray_ContiguousFromObject(pobj, PyArray_DOUBLE, 1, 1); if(!pin) {error....} din = pin->din; The contiguous call makes sure that: a. the object given can be converted into an array of the desired type and dimension b. the data area din is a contiguous field in row-major order. c. if the input data was all that, pin will just be pobj with an incremented refcount. When done with din you should decref pin. -----Original Message----- From: Dave Stinchcombe To: matrix-sig@python.org Date: Monday, March 08, 1999 7:07 AM Subject: [Matrix-SIG] returning arrays >Hi Folks, >it's that time in my work when graphs and charts need to be drawn so I'm >palying with NumPy again. There is one thing I would like to do, but can't >figure out. I would like to write a function which accepts a Numeric.array >as it's argument and returns a Numeric.array, in exactly the kind of way >that Numeric.cos does. Can any one tell me the trick. My function is just a >shortish bit of numerics that requires only one parameter in order to get a >solution. > >Yours >Dave > > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From Oliphant.Travis@mayo.edu Tue Mar 9 22:42:32 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Tue, 9 Mar 1999 16:42:32 -0600 (EST) Subject: [Matrix-SIG] Version 1.1 of cephesmodule now available. Message-ID: For users of the Numerical Extensions to Python who are interested in special function evaluation, version 1.1 of cephesmodule is now available. This version adds the amos libraries from netlib.org to the module so that all of the bessel functions can be called with complex arguments. Functions added: - hankel1 - hankel2 - kv - exponentially scaled versions of all bessel and airy functions. You need a fortran compiler (like g77 or f2c equivalent) to compile this library and module. A gzipped tarfile is available along with glibc based RPMS for Linux at http://oliphant.netpedia.net

cephesmodule 1.1 - An extension module that works with the Numerical Extensions to add many numerical special functions like bessel, elliptic, and hypergeometric, to Python. (09-Mar-99) -------------------------------------------------- Travis Oliphant 200 First St SW Rochester MN 55905 Ultrasound Research Lab (507) 286-5293 Mayo Graduate School Oliphant.Travis@mayo.edu From godzilla@netmeg.net (Les Schaffer) Wed Mar 10 01:50:43 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Tue, 9 Mar 1999 20:50:43 -0500 (EST) Subject: [Matrix-SIG] Re: cephesmodule In-Reply-To: References: Message-ID: <14053.53235.29210.275936@netmeg.net> >>>>> "Travis" == Travis Oliphant writes: Travis> This version adds the amos libraries from netlib.org to Travis> the module so that all of the bessel functions can be Travis> called with complex arguments. hey, this is startin to get real yummy. 1.) how bout spherical harmonics -- they're used all over physics... 2.) since you seem to be cranking up on this, how bout a suggestion: some way for the rest of us moochers to contribute code to your project, so if some yahoo wants spherical harmonics, (s)he can very well cough up C code, SWIG interface, however you're generating your modules. les -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From Oliphant.Travis@mayo.edu Wed Mar 10 04:55:26 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Tue, 9 Mar 1999 22:55:26 -0600 (EST) Subject: [Matrix-SIG] Re: cephesmodule In-Reply-To: <14053.53235.29210.275936@netmeg.net> Message-ID: > 1.) how bout spherical harmonics -- they're used all over physics... > Do you know of some code that computes them? Isn't there someway to write them as a hypergeometric functions? > 2.) since you seem to be cranking up on this, how bout a suggestion: > some way for the rest of us moochers to contribute code to your > project, so if some yahoo wants spherical harmonics, (s)he can very > well cough up C code, SWIG interface, however you're generating your > modules. > Any help would be greatly appreciated. That was part of the reason I wanted to get some documentation out, because I know how confusing it can be at first. If you have a routine that would fit nicely into cephesmodule then just send it to me by email. Any scalar function with any number of inputs and outputs would fit into cephes quite easily. I'm willing to take a spectrum of contributions, from raw code (C or double precision FORTRAN) to a patch against the main sources. For sigtools, there is a need for python code like the wiener filter I posted a few days ago. A colleague of mine will be working on some filter design tools like MATLAB has in the next couple of months. I will be translating some MATLAB code to Python to do Savitsky-Golay filtering on arrays (I use this kind of filter to estimate derivatives in noise.) in the next months. I'm also in need of a nonlinear zero finder. I've found several on the net but need to wrap one up. I'm looking for good ways to organize the functions I'm finding and good names to call the modules, too, if anyone has any idea and/or interest, let me know. (I'm open to changing the names of what I've already released if anyone has a great convention idea.) For most of the modules I have been submitting I have not been using SWIG since I only had a few routines in sigtools and I dealt with the ufuncobject in cephes and it isn't obvious to me how to use SWIG in this case with anymore ease than it was to just write the C code. The ufuncobject is really quite a nice piece of work. If you decide you want to work on something, let me know so that I won't be repeating your work. So, how's that for a great coordinating idea--send me email and/or post to this list. Not to fancy but effective. If anyone has other ideas I'd love to hear them. It would be nice to have a CVS site somewhere but for now sending email and/or posting to this list will work. Best, Travis From smf26@cus.cam.ac.uk Wed Mar 10 19:40:01 1999 From: smf26@cus.cam.ac.uk (Stuart Feerick) Date: Wed, 10 Mar 1999 19:40:01 +0000 (BST) Subject: [Matrix-SIG] QUERY: Array indexing Message-ID: hi: I'm attempting to analyse some simulation data arranged in indexed columns. I'd like to use this to create (the obvious) multi-dimensional array. Indexed by the index values rather than integers I've come up against what I'm sure is a trivial problem, but it's confused me for most of the day. D'oh: I have an index of values, i=[1,2,3], and need to access element [1,2,3] of the array, a=array([3,3,3]). How do I do this going through i? Thanks for any help, stu PS. If there is any interest in an array that can be accessed using values (sort of a multi-dimensional dictionary, I guess), I could post the code when it's in shape. -- ======================================================== Stuart Feerick, Laboratory of Computational Neuroscience The Babraham Institute, Babraham, CAMBRIDGE. CB2 4AT T:01223 496256 F: 01223 496031 E:smf26@cam.ac.uk ======================================================== ~ From janne@avocado.pc.helsinki.fi Wed Mar 10 20:19:08 1999 From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen) Date: 10 Mar 1999 22:19:08 +0200 Subject: [Matrix-SIG] QUERY: Array indexing In-Reply-To: Stuart Feerick's message of "Wed, 10 Mar 1999 19:40:01 +0000 (BST)" References: Message-ID: Stuart Feerick writes: > I have an index of values, i=[1,2,3], and need to access element > [1,2,3] of the array, a=array([3,3,3]). How do I do this going through i? As far as I understand what you want, you can just write a[i]=5, for example, or z=a[i]. the variable i may be a sequence of indexes: a=zeros((2,3,4)) for i in transpose(reshape(indices(a.shape),(len(a.shape),-1))): a[i]=1 There's also the operator module which may allow more flexibility: import operator value1=operator.__getitem__(a,i) operator.__setitem__(a,i,value2) -- Janne From godzilla@netmeg.net (Les Schaffer) Wed Mar 10 21:01:45 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Wed, 10 Mar 1999 16:01:45 -0500 (EST) Subject: [Matrix-SIG] Re: cephesmodule In-Reply-To: References: <14053.53235.29210.275936@netmeg.net> Message-ID: <14054.56761.737709.614302@netmeg.net> > Do you know of some code that computes them? Isn't there someway to > write them as a hypergeometric functions? i wouldnt be surprised, but i doubt its worth it :-) Numerical Recipes has some suggestions for using recurrence relations, which i think i read, somewhere, in this particular case are very fast. i could see coding it up in such a way that it would work great with Python. given all the normalization issues, etc. one thing, a spherical harmonic generally takes two scaler indicies, and two angular positions, which ideally we could use ufuncs on. would ufuncs handle such a thing? say we have a 2D matrix each of theta and phi values; can we call Y( l, m, theta, phi) l,m = integers, theta,phi = 2D matrix of floats and get back a 2D matrix of values? that would be very very cool. les -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From miller5@uiuc.edu Wed Mar 10 22:01:29 1999 From: miller5@uiuc.edu (Mike Miller) Date: 10 Mar 1999 16:01:29 -0600 Subject: [Matrix-SIG] Re: cephesmodule In-Reply-To: Les Schaffer's message of "Wed, 10 Mar 1999 16:01:45 -0500 (EST)" References: <14053.53235.29210.275936@netmeg.net> <14054.56761.737709.614302@netmeg.net> Message-ID: <7690d5uhqe.fsf@uxf.npl.uiuc.edu> >>>>> "Les" == Les Schaffer writes: >> Do you know of some code that computes them? Isn't there >> someway to write them as a hypergeometric functions? #!/usr/bin/env python # # Legendre Polynomials. # # Oct-25-1998 # M.A.Miller, NPL UIUC """ class Legendre: Legendre polynomials. For n = 0-5, they are hard coded: P0(x), P1(x), P2(x), P3(x), P4(x), P5(x). For n > 5, the recurrence relation is used: Pn(n,x). """ class Legendre: def __init__(self): pass def P0(self,x): result = 1.0 return result def P1(self,x): result = x return result def P2(self,x): result = ( 3.0 * x**2 - 1 ) / 2.0 return result def P3(self,x): result = ( 5.0 * x**3 - 3.0 * x ) / 2.0 return result def P4(self,x): result = ( 35.0 * x**4 - 30.0 * x**2 + 3.0 ) / 8.0 return result def P5(self,x): result = ( 63.0 * x**5 - 70.0 * x**3 + 15.0 * x ) / 8.0 return result def Pn(self,n,x): if n == 0: result = self.P0(x) elif n == 1: result = self.P1(x) elif n == 2: result = self.P2(x) elif n == 3: result = self.P3(x) elif n == 4: result = self.P4(x) elif n == 5: result = self.P5(x) else: result= ( ( 2.0 * n - 1.0 ) * x * self.Pn(n-1,x) - ( n - 1.0 ) * self.Pn(n-2,x) ) / n return result def test(): import Numeric L = Legendre() for x in Numeric.arrayrange( -1.0, 1.0, 0.1 ): print '%6.2f %7.4f %7.4f %7.4f %7.4f %7.4f %7.4f %7.4f %7.4f' % \ ( x, L.P0(x), L.P1(x), L.P2(x), L.P3(x), L.P4(x), L.P5(x), L.Pn(5,x), L.Pn(7,x) ) if __name__ == '__main__': test() From managan@llnl.gov Wed Mar 10 22:23:40 1999 From: managan@llnl.gov (Rob Managan) Date: Wed, 10 Mar 1999 14:23:40 -0800 Subject: [Matrix-SIG] Re: cephesmodule In-Reply-To: <7690d5uhqe.fsf@uxf.npl.uiuc.edu> References: Les Schaffer's message of "Wed, 10 Mar 1999 16:01:45 -0500 (EST)" <14053.53235.29210.275936@netmeg.net> <14054.56761.737709.614302@netmeg.net> Message-ID: Probably doesn't make much difference up to P5 but the polynomials should be nested to help avoid cancelation problems. At 4:01 PM -0600 3/10/99, Mike Miller wrote: >>>>>> "Les" == Les Schaffer writes: > > >> Do you know of some code that computes them? Isn't there > >> someway to write them as a hypergeometric functions? > >#!/usr/bin/env python ># ># Legendre Polynomials. ># ># Oct-25-1998 ># M.A.Miller, NPL UIUC > >""" >class Legendre: Legendre polynomials. For n = 0-5, they are hard >coded: P0(x), P1(x), P2(x), P3(x), P4(x), P5(x). For n > 5, the >recurrence relation is used: Pn(n,x). >""" >class Legendre: > def __init__(self): > pass > > def P0(self,x): > result = 1.0 > return result > > def P1(self,x): > result = x > return result > > def P2(self,x): > result = ( 3.0 * x**2 - 1 ) / 2.0 > return result > > def P3(self,x): > result = ( (5.0 * x*x - 3.0) * x ) / 2.0 > return result > > def P4(self,x): > result = ( (35.0 * x*x - 30.0) * x*x + 3.0 ) / 8.0 > return result > > def P5(self,x): > result = ( ((63.0 * x*x - 70.0) * x*x + 15.0) * x ) / 8.0 > return result > > def Pn(self,n,x): > if n == 0: > result = self.P0(x) > elif n == 1: > result = self.P1(x) > elif n == 2: > result = self.P2(x) > elif n == 3: > result = self.P3(x) > elif n == 4: > result = self.P4(x) > elif n == 5: > result = self.P5(x) > else: > result= ( ( 2.0 * n - 1.0 ) * x * self.Pn(n-1,x) - > ( n - 1.0 ) * self.Pn(n-2,x) ) / n > return result > >def test(): > import Numeric > L = Legendre() > for x in Numeric.arrayrange( -1.0, 1.0, 0.1 ): > print '%6.2f %7.4f %7.4f %7.4f %7.4f %7.4f %7.4f %7.4f >%7.4f' % \ > ( x, L.P0(x), L.P1(x), L.P2(x), L.P3(x), L.P4(x), L.P5(x), > L.Pn(5,x), L.Pn(7,x) ) > >if __name__ == '__main__': > test() > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig *-*-*-*-*-*-*-*-*-*-**-*-*-*-*-*-*-*-*-*-*- Rob Managan mailto://managan@llnl.gov LLNL ph: 925-423-0903 P.O. Box 808, L-098 FAX: 925-423-5804 Livermore, CA 94551-0808 From pas@scansoft.com Thu Mar 11 02:29:17 1999 From: pas@scansoft.com (Perry Stoll) Date: Wed, 10 Mar 1999 18:29:17 -0800 Subject: [Matrix-SIG] QUERY: Array indexing Message-ID: <002b01be6b66$f62be2a0$374cf60d@kyoto.xis.xerox.com> Assuming you are using NumPy, the function you want is Numeric.take(). See: http://starship.python.net/crew/da/numtut/array.html#SEC24 And from the incomparable Tim Peters, here's a lesson on gather/scatter in general (sorry if this link gets broken up across two lines): http://x9.dejanews.com/[ST_rn=ps]/getdoc.xp?AN=227662160&CONTEXT=921108267.1 22880047&hitnum=3: which yields the following functions: def gather(seq, indices): from operator import __getitem__ return map(__getitem__, (seq,)*len(indices), indices) def scatter(seq, indices,vals): from operator import __setitem__ return map(__setitem__, (seq,)*len(indices), indices,vals) Cheers, Perry -----Original Message----- From: Janne Sinkkonen To: Stuart Feerick Cc: matrix-sig@python.org Date: Wednesday, March 10, 1999 12:48 PM Subject: Re: [Matrix-SIG] QUERY: Array indexing >Stuart Feerick writes: > >> I have an index of values, i=[1,2,3], and need to access element >> [1,2,3] of the array, a=array([3,3,3]). How do I do this going through i? > >As far as I understand what you want, you can just write a[i]=5, for >example, or z=a[i]. the variable i may be a sequence of indexes: > >a=zeros((2,3,4)) >for i in transpose(reshape(indices(a.shape),(len(a.shape),-1))): a[i]=1 > From frank@ned.dem.csiro.au Thu Mar 11 06:46:01 1999 From: frank@ned.dem.csiro.au (Frank Horowitz) Date: Thu, 11 Mar 1999 14:46:01 +0800 Subject: [Matrix-SIG] QUERY: Array indexing In-Reply-To: <002b01be6b66$f62be2a0$374cf60d@kyoto.xis.xerox.com> Message-ID: At 10:29 AM +0800 11/3/99, Perry Stoll wrote: >And from the incomparable Tim Peters, here's a lesson on gather/scatter in >general (sorry if this link gets broken up across two lines): > >http://x9.dejanews.com/[ST_rn=ps]/getdoc.xp?AN=227662160&CONTEXT=921108267.1 >22880047&hitnum=3: > >which yields the following functions: > >def gather(seq, indices): > from operator import __getitem__ > return map(__getitem__, (seq,)*len(indices), indices) > >def scatter(seq, indices,vals): > from operator import __setitem__ > return map(__setitem__, (seq,)*len(indices), indices,vals) > Thanks for that "incomparable Tim Peters" citation; that's a fascinating thread (have I sucked up enough yet? :-). My applications require local references in the array (somewhat akin to finite difference/convolution templates) but the operations I need to perform are more general than multiply-and-sum, hence I can't use the recent n-d convolve routines from Travis Oliphant (or was it someone else; senility is obviously setting in...) My approach to date has been to build shifted/rolled versions of the array, and operate with them (code available upon request, if anyone is silly enough to want it :-). Obviously, this comes at a severe cost in memory (hey! I'm trying to do this in 3d with ~2k X ~2k base images, and a couple of "+/- Z" steps for the image, computable from theory). As you might expect I start thrashing virtual memory PDQ. It seems to me that something like this gather operation would do the trick, with one hell of a lot less memory thrashing, but I can't quite fathom the idiom for generating an indexing sequence that says something like "one over in the plus X direction at the same time as one over in the negative Z direction" . I'm sure I'm missing something obvious, but I *am* missing it. Wisdom anyone? (Tim? :-) TIA, Frank Horowitz -- Frank Horowitz frank@ned.dem.csiro.au Australian Geodynamics Cooperative Research Centre, and CSIRO-Exploration & Mining, PO Box 437, Nedlands, WA 6009, AUSTRALIA Direct: +61 8 9284 8431; FAX: +61 8 9389 1906; Reception: +61 8 9389 8421 From edcjones@erols.com Wed Mar 10 10:25:23 1999 From: edcjones@erols.com (Edward C. Jones) Date: Wed, 10 Mar 1999 05:25:23 -0500 Subject: [Matrix-SIG] Accessing raw data in PyArrays from C Message-ID: <36E64892.F144C752@erols.com> I have been writing some C code that extends NumPy. I find myself regularly using the C macro: #define ARRAYDATUM(type,charptr) (*( ((type) *) (charptr))) to access the data in the array. For example, suppose "pao" is a pointer to a PyArrayObject holding unsigned chars. Then ARRAYDATUM(unsigned char, pao->data) returns the first byte of data in the array. But this macro assumes I know the type of the array. In arraytypes.c, there are functions like (using UBYTE as an example): static PyObject * UBYTE_getitem(char *ip) {return PyInt_FromLong((long)*((unsigned char *)ip));} static int UBYTE_setitem(PyObject *op, char *ov) {*((unsigned char *)ov)=PyInt_AsLong(op);return PyErr_Occurred() ? -1:0;} I suggest that something like the following be added: static unsigned byte UBYTE_getraw(char *ip) {return *((unsigned char *)ip);} static int UBYTE_setraw(unsigned cahr b, char *ov) {*((unsigned char *)ov)=b;} How can this be done? Thanks, Ed Jones From hinsen@cnrs-orleans.fr Thu Mar 11 15:26:48 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 11 Mar 1999 16:26:48 +0100 Subject: [Matrix-SIG] QUERY: Array indexing In-Reply-To: (message from Stuart Feerick on Wed, 10 Mar 1999 19:40:01 +0000 (BST)) References: Message-ID: <199903111526.QAA24384@dirac.cnrs-orleans.fr> > I have an index of values, i=[1,2,3], and need to access element > [1,2,3] of the array, a=array([3,3,3]). How do I do this going through i? What do you mean by "element [1,2,3]"? If you want *one* element, this should be three indices correspdonding to three dimensions - but the array you give is one-dimensional! -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From warren@pfiesteria.gsfc.nasa.gov Thu Mar 11 23:37:24 1999 From: warren@pfiesteria.gsfc.nasa.gov (Warren B. Focke) Date: Thu, 11 Mar 1999 18:37:24 -0500 (EST) Subject: [Matrix-SIG] buglet and fix Message-ID: > >>> from Numeric import array > >>> a=array((1,2,3)) > >>> del a[2:] > Segmentation fault The fix for this (in array_ass_slice) is copied directly from array_ass_sub. I suspect it should go in array_ass_item too. I couldn't activate that path from within the interpreter, but an extension module could call PyObject_DelItem on an array. Warren Focke --- arrayobject.c.orig Wed Mar 10 23:20:43 1999 +++ arrayobject.c Wed Mar 10 23:12:30 1999 @@ -520,6 +520,11 @@ int ret; PyArrayObject *tmp; + if (v == NULL) { + PyErr_SetString(PyExc_ValueError, "Can't delete array elements."); + return -1; + } + if ((tmp = (PyArrayObject *)array_slice(self, ilow, ihigh)) == NULL) return -1; ret = PyArray_CopyObject(tmp, v); Py_DECREF(tmp); From dubois1@llnl.gov Thu Mar 11 23:57:06 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Thu, 11 Mar 1999 15:57:06 -0800 Subject: [Matrix-SIG] Re: [C++-SIG] CXX (Release 8) and EGCS 1.1.1 Message-ID: <001f01be6c1a$de7df9e0$f4160218@c1004579-c.plstn1.sfba.home.com> I have now learned enough Linux that I installed EGCS 1.1.1 and tried to compile CXX, but even with some corrections to deal with the std:: problem and the error you got, I still get internal compiler errors. I will look into this some more. One correspondent cleverly worked around the errors but broke the code by removing a virtual behavior. Since the error is in a virtual function in a templated class, it is at least believable that there is a compiler error. g++ -fpic -I./Include -I./Demo -g -I/usr/local/beta/dubois/stockpython/inc lude/python1.5 -I/usr/local/beta/dubois/stockpython/include/python1.5 -DHAVE _CONFIG_H -c ./Demo/r.cxx Include/CXX_Objects.h: In method `int ::Py::String::capacity() const': In file included from Include/CXX_Extensions.h:8, from Demo/r.h:3, from ./Demo/r.cxx:1: Include/CXX_Objects.h:1237: Internal compiler error. -----Original Message----- From: John Barnard To: c++-sig@python.org Date: Wednesday, January 13, 1999 3:43 PM Subject: [C++-SIG] CXX (Release 8) and EGCS 1.1.1 >Has anyone successfully used CXX (Release 8) with EGCS 1.1.1? When I >try to compile any of the examples that are distributed with CXX I get >the following messages: > >gcc -I/home/barnard/lib/include -I/usr/include/python1.5 arraytest.cxx >/home/barnard/lib/include/CXX_Objects.h: In method `class ::Py::seqref ::Py::SeqBase::front()': >In file included from /home/barnard/lib/include/CXX_Array.h:3, > from arraytest.cxx:5: >/home/barnard/lib/include/CXX_Objects.h:923: use of class template `template ::Py::seqref' as expression >/home/barnard/lib/include/CXX_Objects.h: In method `class ::Py::seqref ::Py::SeqBase::back()': >/home/barnard/lib/include/CXX_Objects.h:931: use of class template `template ::Py::seqref' as expression >/home/barnard/lib/include/CXX_Objects.h: In method `int ::Py::String::capacity() const': >/home/barnard/lib/include/CXX_Objects.h:1229: confused by earlier errors, bailing out >make: *** [all] Error 1 > >I'm not sure if the problem is with the template mechanism or the >namespace mechanism (everything that's supposed to be under the std >namespace is in the global namespace for the libstdc++ distributed >with EGCS 1.1.1). Turning off namespace support in CXX doesn't help. > >I'm running Red Hat 5.2 (2.0.36) on a Pentium II. > >Thanks, > >John Barnard >Assistant Professor >Department of Statistics >Harvard University >Phone: (617) 495-1603 >Fax: (617) 496-8057 >Email: barnard@stat.harvard.edu > >_______________________________________________ >C++-SIG maillist - C++-SIG@python.org >http://www.python.org/mailman/listinfo/c++-sig > > From tim_one@email.msn.com Fri Mar 12 07:29:13 1999 From: tim_one@email.msn.com (Tim Peters) Date: Fri, 12 Mar 1999 02:29:13 -0500 Subject: [Matrix-SIG] QUERY: Array indexing In-Reply-To: Message-ID: <000201be6c5a$070b7380$a19e2299@tim> [Perry Stoll, locates an old timbot gather/scatter tutorial] [Frank Horowitz chews & chews, but is still hungry] > Thanks for that "incomparable Tim Peters" citation; that's a fascinating > thread (have I sucked up enough yet? :-). No. Keep going! We'll let you know when it's enough . > My applications require local references in the array (somewhat akin to > finite difference/convolution templates) but the operations I need to > perform are more general than multiply-and-sum, hence I can't use the > recent n-d convolve routines from Travis Oliphant (or was it someone > else; senility is obviously setting in...) > > My approach to date has been to build shifted/rolled versions of the > array, and operate with them (code available upon request, if anyone is > silly enough to want it :-). Obviously, this comes at a severe cost in > memory ... I don't know the best way to approach this in NumPy, perhaps because I haven't yet installed it . In straight Python I do stuff like this by writing thin wrapper classes that leave the original matrices alone, merely permuting the *indices* in __setitem__/__getitem__ (instead of physically permuting the *data*). For example, if your need to transpose a 2D array, just swap the indices in the wrappers before passing them on to the actual (still untransposed) array. Here's an overly simplistic but complete example: class VectorRotate: def __init__(self, data, shift): """Make data act as if it had been rotated left by shift slots""" self.x = data self.n = len(data) self.shift = shift def __getitem__(self, i): return self.x[(i + self.shift) % self.n] def __setitem__(self, i, val): self.x[(i + self.shift) % self.n] = val def __len__(self): return self.n def getdata(self): return self.x vec = range(10) print vec r = VectorRotate(vec, 3) r[0] = "hmm!" for i in range(len(r)): print i, r[i] print r.getdata() That prints: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 0 hmm! 1 4 2 5 3 6 4 7 5 8 6 9 7 0 8 1 9 2 [0, 1, 2, 'hmm!', 4, 5, 6, 7, 8, 9] If you want to get really fancy, you can define arbitrarily complex N-dimensional "mapping objects" independent of specific matrices, and manipulate them for their own sake, reluctantly applying them to actual matrices when deadlines get too close . > It seems to me that something like this gather operation would do the > trick, with one hell of a lot less memory thrashing, but I can't quite > fathom the idiom for generating an indexing sequence that says something > like "one over in the plus X direction at the same time as one over in the > negative Z direction" . If the above was suggestive enough, I expect you'll find this kind of thing easier than gather/scatter. Yes? > ... > Wisdom anyone? (Tim? :-) Sorry, you didn't suck up enough to get a wise answer! although-you're-off-to-a-fine-start-ly y'rs - tim From Oliphant.Travis@mayo.edu Fri Mar 12 08:15:48 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 12 Mar 1999 02:15:48 -0600 (EST) Subject: [Matrix-SIG] C Extensions with Numeric Message-ID: I've just updated my informal material for someone trying to write Extensions modules for NumPy. I now include information regarding reference counting which must be handled correctly in any extensions module. This was a glaring hole in what I produced previously. Gzipped pdf, ps, and tex sources are at http://oliphant.netpedia.net From frank@ned.dem.csiro.au Fri Mar 12 10:25:35 1999 From: frank@ned.dem.csiro.au (Frank Horowitz) Date: Fri, 12 Mar 1999 18:25:35 +0800 Subject: [Matrix-SIG] Shifted array ops (was: RE: [Matrix-SIG] QUERY: Array indexing) In-Reply-To: <000201be6c5a$070b7380$a19e2299@tim> References: Message-ID: At 3:29 PM +0800 12/3/99, Tim Peters wrote: >[Frank Horowitz chews & chews, but is still hungry] >> My applications require local references in the array (somewhat akin to >> finite difference/convolution templates) but the operations I need to >> perform are more general than multiply-and-sum, hence I can't use the >> recent n-d convolve routines from Travis Oliphant (or was it someone >> else; senility is obviously setting in...) >> >> My approach to date has been to build shifted/rolled versions of the >> array, and operate with them (code available upon request, if anyone is >> silly enough to want it :-). Obviously, this comes at a severe cost in >> memory ... > >I don't know the best way to approach this in NumPy, perhaps because I >haven't yet installed it . In straight Python I do stuff like this by >writing thin wrapper classes that leave the original matrices alone, merely >permuting the *indices* in __setitem__/__getitem__ (instead of physically >permuting the *data*). For example, if your need to transpose a 2D array, >just swap the indices in the wrappers before passing them on to the actual >(still untransposed) array. > >Here's an overly simplistic but complete example: >class VectorRotate: > def __init__(self, data, shift): > """Make data act as if it had been rotated left by shift slots""" > self.x = data > self.n = len(data) > self.shift = shift > > def __getitem__(self, i): > return self.x[(i + self.shift) % self.n] > > def __setitem__(self, i, val): > self.x[(i + self.shift) % self.n] = val > > def __len__(self): > return self.n > > def getdata(self): > return self.x > >If the above was suggestive enough, I expect you'll find this kind of thing >easier than gather/scatter. Yes? Err, I'm still not sure I quite understand (more chewing sounds evident in the background; do I feel like a cow, or what? :-). If I understand your suggestion correctly, you're doing some trickery with indexing *for an individual element* (i.e. your __getitem__ __setitem__ routines index a specific element). Does the fact that they're overriding the operator class methods imply that the result of the calls to getdata index through the array at C speeds (courtesy of the NumPy __getitem__ support code)? Or is the result that the getdata indexes through the array at interpreter speeds? Remember, I'm contemplating applying this stuff (over and over :-) to arrays with ~12 million elements. If it runs at interpreter speeds, I'll have to look for a different answer. > >> ... >> Wisdom anyone? (Tim? :-) > >Sorry, you didn't suck up enough to get a wise answer! > OK, then thanks for the half-wise answer! (And I grovel in your general direction, which I believe must more or less be straight down from here in Perth :-) Cheers, Frank Horowitz -- Frank Horowitz frank@ned.dem.csiro.au Australian Geodynamics Cooperative Research Centre, and CSIRO-Exploration & Mining, PO Box 437, Nedlands, WA 6009, AUSTRALIA Direct: +61 8 9284 8431; FAX: +61 8 9389 1906; Reception: +61 8 9389 8421 From pas@xis.xerox.com Fri Mar 12 15:45:11 1999 From: pas@xis.xerox.com (Perry Stoll) Date: Fri, 12 Mar 1999 10:45:11 -0500 Subject: [Matrix-SIG] Shifted array ops (was: RE: [Matrix-SIG] QUERY: Array indexing) Message-ID: <004201be6c9f$5d660700$4a4df60d@nara.xis.xerox.com> [Tim Peter's continues to dazzle, but stays true to his Buddha-nature by avoiding a practical solution ;] What I understand you're asking for is to perform operations on shifted/rotated versions of big matrices (aka numeric arrays). If that's what you're after, there is a way to do this sort of thing in NumPy using the built-in (since Python 1.5, I believe) slice type. The Python deities, in their glorious munificence, have given us the built-in function slice() to create slice objects. The idea is to build up a slice object to represent the particular view of the data needed and then apply the slice object to the numeric array you have. The slicing operation doesn't create a copy of the data; it's just a new view of the original data. You'll need to learn how slice objects get interpreted by Numeric, but I don't know where that's documented other than in the source. Does anyone know if such documetation exists? As a concrete example, take look at the functions in http://starship.python.net/crew/jhauser/NumAdd.py.html Here's the function to compute a central diffence using this technique: def diff(m,axis=0): """ Foward difference of m along axis axis. """ if m.shape[axis] < 2: raise 'Error, axis needs at least be of length 2' l_sl=[slice(None,None,None)]*len(m.shape) u_sl=l_sl[:] l_sl[axis]=slice(1,None,1) u_sl[axis]=slice(None,-1,1) return m[l_sl]-m[u_sl] Another good place to find this style of index-building is in the FancyArray.__{get,set}item__ method at http://starship.skyport.net/~hochberg/FancyArray.py.html Of course, getting the end-cases right is your job ;) -Perry From dubois1@llnl.gov Fri Mar 12 16:13:02 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 12 Mar 1999 08:13:02 -0800 Subject: [Matrix-SIG] Shifted array ops (was: RE: [Matrix-SIG] QUERY: Array indexing) Message-ID: <001101be6ca3$346ed3a0$f4160218@c1004579-c.plstn1.sfba.home.com> >You'll need to learn how slice objects get interpreted by Numeric, but I >don't know where that's documented other than in the source. Does anyone know >if such documetation exists? > David Ascher is finishing up the first draft of the Numerical documentation, which as I understand it will have the reference, tutorial, and C-API documentation with help from Konrad and Travis. As soon as he gets it to me and I get the legal boilerplate on it I will slap it up where everyone can get it. With feedback from the community we will then have an answer to your question. I won't try to give an ETA, but he sounded like he was the proverbial week away recently. From hinsen@cnrs-orleans.fr Fri Mar 12 19:07:02 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 12 Mar 1999 20:07:02 +0100 Subject: [Matrix-SIG] Shifted array ops (was: RE: [Matrix-SIG] QUERY: Array indexing) In-Reply-To: <004201be6c9f$5d660700$4a4df60d@nara.xis.xerox.com> (pas@xis.xerox.com) References: <004201be6c9f$5d660700$4a4df60d@nara.xis.xerox.com> Message-ID: <199903121907.UAA19262@dirac.cnrs-orleans.fr> > Another good place to find this style of index-building is in the > FancyArray.__{get,set}item__ method at > http://starship.skyport.net/~hochberg/FancyArray.py.html Also in the module Interpolation.py in my "scientific Python" collection (http://starship.python.net/crew/hinsen/scientific.html). -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From hinsen@cnrs-orleans.fr Fri Mar 12 19:08:21 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 12 Mar 1999 20:08:21 +0100 Subject: [Matrix-SIG] Accessing raw data in PyArrays from C In-Reply-To: <36E64892.F144C752@erols.com> (edcjones@erols.com) References: <36E64892.F144C752@erols.com> Message-ID: <199903121908.UAA10626@dirac.cnrs-orleans.fr> > I have been writing some C code that extends NumPy. I find myself > regularly using the C macro: > > #define ARRAYDATUM(type,charptr) (*( ((type) *) (charptr))) Looks fine to me. Why do you want to replace it by functions with added function call overhead? -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From vanandel@atd.ucar.edu Fri Mar 12 21:02:03 1999 From: vanandel@atd.ucar.edu (Joe Van Andel) Date: Fri, 12 Mar 1999 14:02:03 -0700 Subject: [Matrix-SIG] Re: [C++-SIG] CXX (Release 8) and EGCS 1.1.1 In-Reply-To: Your message of Fri, 12 Mar 1999 12:01:13 -0500. <199903121701.MAA00575@python.org> Message-ID: <199903122102.OAA27317@stout.atd.ucar.edu> Reply-To: vanandel@ucar.edu Cc: Fcc: -------- Regarding the template and namespace problems with CXX Release 8 and EGCS 1.1.1 I had reported this problem as bug report to the EGCS mailing list. I've just heard from one of the EGCS developers and he tells that egcs-2.93.11 (i.e. the current snapshot) will compile the source with minor changes. These changes will appear in EGCS 1.2 (not 1.1.2, which may be out today.) martin@mira.isdn.cs.tu-berlin.de said: > There is a simple work-around: Instead of writing > std ::iterator< std ::random_access_iterator_tag, seqref, int> > you can write > random_access_iterator< seqref, int> Joe VanAndel Internet: vanandel@ucar.edu National Center for Web: http://www.atd.ucar.edu/~vanandel/home.html Atmospheric Research From frank@ned.dem.csiro.au Sat Mar 13 10:21:50 1999 From: frank@ned.dem.csiro.au (Frank Horowitz) Date: Sat, 13 Mar 1999 18:21:50 +0800 Subject: [Matrix-SIG] Shifted array ops (was: RE: [Matrix-SIG] QUERY: Array indexing) References: <004201be6c9f$5d660700$4a4df60d@nara.xis.xerox.com> Message-ID: <36EA3C3E.2C4D8EB1@ned.dem.csiro.au> Perry Stoll wrote: > > [Tim Peter's continues to dazzle, but stays true to his Buddha-nature by > avoiding a practical solution ;] > > What I understand you're asking for is to perform operations on > shifted/rotated versions of big matrices (aka numeric arrays). If that's what > you're after, there is a way to do this sort of thing in NumPy using the > built-in (since Python 1.5, I believe) slice type. The Python deities, in > their glorious munificence, have given us the built-in function slice() > to create slice objects. > > The idea is to build up a slice object to represent the particular view of > the data needed and then apply the slice object to the numeric array you > have. The slicing operation doesn't create a copy of the data; it's just a > new view of the original data. > > You'll need to learn how slice objects get interpreted by Numeric, but I > don't know where that's documented other than in the source. Does anyone know > if such documetation exists? > > As a concrete example, take look at the functions in > http://starship.python.net/crew/jhauser/NumAdd.py.html > [A clinking sound, as the penny *finally* drops for Frank.] BINGO! > Of course, getting the end-cases right is your job ;) > Geee. And here I thought that posting dumb questions to this list was my job :-) Thanks to all concerned for their helpful answers! (We now return you to your regularly scheduled "dazzling" soliloquys ;) From Oliphant.Travis@mayo.edu Sat Mar 13 23:28:24 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Sat, 13 Mar 1999 17:28:24 -0600 (EST) Subject: [Matrix-SIG] Re: cephesmodule In-Reply-To: <14058.59758.305300.21469@netmeg.net> Message-ID: > > >> It would handle it superbly, this is exactly what ufuncs are > >> for. > i'm gonna have to lok through the ufunc docs and internals just to see > how it all works, and how to write extensions which use them. it > sounds great, > > >> But it could go beyond this. With a single statement you could > >> evaluate Y(l,m,theta,phi) for all l and m up to some limit > >> where for each l and m the function is evaluated over matrices > >> theata and phi. This bit of magic is accomplished using the > >> broadcasting rules that are builtin to ufunc objects. > > okay. i am starting to think about how i wanna code the spherical > harmonics stuff. one thing i need which i dont see: a factorial > function. it'd be nice to have one (rather than using the gamma > function, say), since lots of the series solutions type of stuff i > gonna use them. and its a no brainer to code up? would you wanna add > one to the cephes module, or should i see if guido would want to add > it to the math library in pythn, or perhaps into NumPy math. > > les You know I think there are a lot of things that don't get coded up formally because they are so easy to produce. For example: def fact(n): return multiply.reduce(arange(1,n)) This of course is not a ufunc which means it can't be called with an array of numbers. You know it would be really easy for me to add it to the cephes library. I don't know what to do about overflow though, as the question of handling errors in the cephes module is still up in the air. Another question is developing a library of functions that are implemented in Python. So far cephes is all C or Fortran code. I'm thinking it would be good to have a Python wrapper around the module that defined some functions in Python. I would want all of these functions to be built around ufuncs though so that the caller could count on broadcasting and array handling. Perhaps a separate Python-only module should be developed? Are you thinking of writing your spherical harmonic code in Python or C? I was poking around in MATLAB and noticed that their spherical harmonics (associated legendre function) function legendre.m is written in MATLAB and it handles complex numbers. I've not heard of an application of associated Legendre polynomials evaluated over the complex field, but would it be useful to have? -Travis From siopis@astro.ufl.edu Sun Mar 14 01:16:25 1999 From: siopis@astro.ufl.edu (Christos Siopis) Date: Sat, 13 Mar 1999 20:16:25 -0500 (EST) Subject: [Matrix-SIG] Casts and slices In-Reply-To: Message-ID: Greetings, fellow NumPyers First off, thanks to all the noble souls who created NumPy and then shared it with the rest of us :) Having just changed from an environment where unlimited IDL (in fact, PV WAVE) licenses were available to a place where no IDL is available, I did appreciate NumPy's availability :) I have a question about what might be a possible inconsistency: >>> b = reshape(ones(20, 'f'), (10,2)) >>> b[:,0] = b[:,1] # This works fine: both sides are of the same type. >>> b[:,0] = sqrt(b[:,1]) # This also works fine. >>> b[:,0] = b[:,1]**3 # Should this not also work if sqrt works? Traceback (innermost last): File "", line 1, in ? TypeError: Array can not be safely cast to required type If the previous does not work because it attempts to cast a double (right hand side) into a float, then should the following not also produce an error, since each element of the list is a Python double: >>> b[:,0] = (b[:,1]**3).tolist() # It works. >>> b[:,0] = map(float, b[:,1]**3) # It also works. Since many operations have right hand sides which are doubles, I was wondering if there is an easier way to assign results to float array elements than using the tolist() method or map(). ==== A second question: using take() one can access noncontiguous elements of an array. Is there a way to also assign values to noncontiguous elements? I mean something like: b[(0,1,4)] = [1.2, 3.4, 5.3] One last question is also coming under a different subject. Thanks, Christos Siopis From siopis@astro.ufl.edu Sun Mar 14 01:47:09 1999 From: siopis@astro.ufl.edu (Christos Siopis) Date: Sat, 13 Mar 1999 20:47:09 -0500 (EST) Subject: [Matrix-SIG] Binary file I/O In-Reply-To: Message-ID: OK, here is the last question: One of the first things I needed with NumPy was binary file I/O. I found Travis' numpyio module, but like he says, some of numpyio's functionality is already supported by Python itself, with the disadvantage that a copy of the array has to be made (e.g., using fromstring() and tostring()). Is this also the case when one uses the built-in array module instead? For example, to read 10 float elements from an open binary file object f: >>> import Numeric, array >>> >>> a=array.array('f') >>> a.fromfile(f, 10) >>> b = Numeric.array(a, copy=0) Will the last line create a new copy of the array in memory much like it would create a new copy from a string? Or is it that the array.array and Numeric.array objects are "related enough" that no copy has to be made? In any case, I was wondering if there is any thought of adding fromfile() and tofile() methods for Numeric arrays (perhaps based on Travis' numpyio?). I saw this mentioned in James Hugunin's very first email posting to this list, but somewhere along the way it must have been dropped. In David Ascher's 1996 NumPy tutorial, it is also suggested using pickle, but this is again wasteful for large arrays and also the resulting files cannot be read from inside other programs. I also have a minor suggestion for numpyio, which would also apply to fromfile()/tofile() if they would ever to be implemented. Why not make the methods more NumPy-friendly by specifying the array to be read or writen using a shape tuple instead of using C library's "number of elements" and "size of element" arguments? I mean something like: from Numeric import * def rbf(f, shape = (-1,), typecode = 'f'): return reshape(fromstring(f.read(product(shape)), typecode), shape) where rbf = "read binary file", and f is an open file object. The effect of the above is: 1. If the shape tuple contains no negative numbers, then nelem = product(shape) elements of requested typecode are read and an array of the appropriate shape is returned. 2. If the shape tuple contains a negative number, then the entire binary file is read and returned in the requested shape. The file's size would have to be a multiple of the product of the shape tuple's nonegative elements or else rehaps() issues an exception. Thanks, Christos From tim_one@email.msn.com Mon Mar 15 03:47:36 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sun, 14 Mar 1999 22:47:36 -0500 Subject: [Matrix-SIG] RE: Shifted array ops (was: RE: [Matrix-SIG] QUERY: Array indexing) In-Reply-To: Message-ID: <000801be6e96$90a2f120$e39e2299@tim> [Frank Horowitz has an "aha!" experience thanks to Perry Stoll (thanks. Perry Stoll!)] [Tim's toy VectorRotate class in non-NumPy Python] [Frank] > If I understand your suggestion correctly, you're doing some trickery > with indexing *for an individual element* ... Exactly, and in non-NumPy Python that doesn't generalize to whole-array operations. > Does the fact that they're overriding the operator class methods imply > that the result of the calls to getdata index through the array at C > speeds (courtesy of the NumPy __getitem__ support code)? Or is the > result that the getdata indexes through the array at interpreter speeds? I *still* haven't installed NumPy , but my non-NumPy example class works at interpreter speed. Presumably combining NumPy slice objects with NumPy whole-array ops works at C speed, though. Same idea under the covers: don't copy the data, apply a transformation to the indices. Simple things like that sure do go faster in C! > Remember, I'm contemplating applying this stuff (over and over :-) to > arrays with ~12 million elements. If it runs at interpreter speeds, I'll > have to look for a different answer. That part was unclear, since you had mentioned disk thrashing. A gimmick that can stop *that* can afford to waste a lot of CPU and still win in the end. That's what I use it for (in non-NumPy Python). As with all things Python, though, you don't have to choose because everything's perfect . although-perfection-is-often-cleverly-hiding-in-plain-view-ly y'rs - tim From jhauser@ifm.uni-kiel.de Mon Mar 15 10:43:43 1999 From: jhauser@ifm.uni-kiel.de (jhauser@ifm.uni-kiel.de) Date: Mon, 15 Mar 1999 11:43:43 +0100 (CET) Subject: [Matrix-SIG] Casts and slices In-Reply-To: References: Message-ID: <14060.57994.966673.405870@lisboa.ifm.uni-kiel.de> Christos Siopis writes: > Greetings, fellow NumPyers > > First off, thanks to all the noble souls who created NumPy and > then shared it with the rest of us :) Having just changed from > an environment where unlimited IDL (in fact, PV WAVE) licenses > were available to a place where no IDL is available, I did > appreciate NumPy's availability :) > > I have a question about what might be a possible inconsistency: > > >>> b = reshape(ones(20, 'f'), (10,2)) > > >>> b[:,0] = b[:,1] # This works fine: both sides are of the same type. > > >>> b[:,0] = sqrt(b[:,1]) # This also works fine. > > >>> b[:,0] = b[:,1]**3 # Should this not also work if sqrt works? > Traceback (innermost last): > File "", line 1, in ? > TypeError: Array can not be safely cast to required type > I don't know, if this is an error, but to be on the save side use: b[:,0] = (b[:,1]**3).astype('f') Not sure if this generates an copy in between, but it's probably faster than map(). > If the previous does not work because it attempts to cast a > double (right hand side) into a float, then should the following > not also produce an error, since each element of the list is > a Python double: > > >>> b[:,0] = (b[:,1]**3).tolist() # It works. > > >>> b[:,0] = map(float, b[:,1]**3) # It also works. > > Since many operations have right hand sides which are doubles, > I was wondering if there is an easier way to assign results to > float array elements than using the tolist() method or map(). See above... > ==== > > A second question: using take() one can access noncontiguous > elements of an array. Is there a way to also assign values to > noncontiguous elements? I mean something like: > > b[(0,1,4)] = [1.2, 3.4, 5.3] > Use the arrayfns module from the graphics package. Althoug it works only on 1-d arrays. >>> from arrayfns import array_set >>> array_set(ravel(b),[0,1,4],[1.2, 3.4, 5.3]) >>> b # now the indices were wrong >>> array([[ 1.20000005, 3.4000001 ], [ 1. , 1. ], [ 5.30000019, 1. ], [ 1. , 1. ], [ 1. , 1. ], [ 1. , 1. ], [ 1. , 1. ], [ 1. , 1. ], [ 1. , 1. ], [ 1. , 1. ]],'f') There was a discussion in the SIG and a proposal by David Ascher for a more general solution. Hope this helps, __Janko From janne@avocado.pc.helsinki.fi Mon Mar 15 14:59:31 1999 From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen) Date: 15 Mar 1999 16:59:31 +0200 Subject: [Matrix-SIG] Nonlinear optimization routines anyone? Message-ID: Hi, does anybody has neatly packaged nonlinear optimization routines implemented in NumPy? I'd like to have either conjugate gradients or BFGS. Explicitly calculating the Hessian is out of question (because of the size and complexity of the problem). I thought translating part of the matlab codes of C. T. Kelley (http://www4.ncsu.edu/eos/users/c/ctkelley/www/matlab_darts.html) to Numerical Python unless something already implemented emerges. -- Janne From da@ski.org Mon Mar 15 17:33:58 1999 From: da@ski.org (David Ascher) Date: Mon, 15 Mar 1999 09:33:58 -0800 (Pacific Standard Time) Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: Message-ID: > does anybody has neatly packaged nonlinear optimization routines > implemented in NumPy? I'd like to have either conjugate gradients or > BFGS. Explicitly calculating the Hessian is out of question (because > of the size and complexity of the problem). > > I thought translating part of the matlab codes of C. T. Kelley > (http://www4.ncsu.edu/eos/users/c/ctkelley/www/matlab_darts.html) to > Numerical Python unless something already implemented emerges. I've never found a routine to package, and I've looked a fair bit. There's COOOL (http://timna.mines.edu/cwpcodes/coool/) which looked interesting, but it seemed extremely non-portable (couldn't get it to compile on Win32, requires expect, etc. etc.). --david From phil@geog.ubc.ca Mon Mar 15 18:03:36 1999 From: phil@geog.ubc.ca (Phil Austin) Date: Mon, 15 Mar 1999 10:03:36 -0800 Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: References: Message-ID: <199903151803.KAA10975@brant.geog.ubc.ca> > > I've never found a routine to package, and I've looked a fair bit. There's > COOOL (http://timna.mines.edu/cwpcodes/coool/) which looked interesting, > but it seemed extremely non-portable (couldn't get it to compile on Win32, > requires expect, etc. etc.). > > --david and it's also been frozen at version 0.1 for the last 3 years or so. Another possibility is MTL, which has support for iterative solvers and has a new version announced this weekend. It would be straightforward to pass MTL arrays to Python using Paul's CXX_Array.h. http://www.lsc.nd.edu/research/mtl/doc/index.html#itl Regards, Phil From dubois1@llnl.gov Mon Mar 15 20:02:57 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Mon, 15 Mar 1999 12:02:57 -0800 Subject: [Matrix-SIG] Nonlinear optimization routines anyone? Message-ID: <002001be6f1e$d20e3100$f4160218@c1004579-c.plstn1.sfba.home.com> The conjugate gradient algorithm is probably about twenty lines or less of matrix/vector statements in Python, assuming you have a preconditioner you can express that way. So just code it up in Python. It will be fast enough, all the hard work is in the dot products and matrix multiplies. ----Original Message----- From: David Ascher To: Janne Sinkkonen Cc: matrix-sig@python.org Date: Monday, March 15, 1999 9:46 AM Subject: Re: [Matrix-SIG] Nonlinear optimization routines anyone? >> does anybody has neatly packaged nonlinear optimization routines >> implemented in NumPy? I'd like to have either conjugate gradients or >> BFGS. Explicitly calculating the Hessian is out of question (because >> of the size and complexity of the problem). >> >> I thought translating part of the matlab codes of C. T. Kelley >> (http://www4.ncsu.edu/eos/users/c/ctkelley/www/matlab_darts.html) to >> Numerical Python unless something already implemented emerges. > >I've never found a routine to package, and I've looked a fair bit. There's >COOOL (http://timna.mines.edu/cwpcodes/coool/) which looked interesting, >but it seemed extremely non-portable (couldn't get it to compile on Win32, >requires expect, etc. etc.). > >--david > > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From da@ski.org Mon Mar 15 20:10:07 1999 From: da@ski.org (David Ascher) Date: Mon, 15 Mar 1999 12:10:07 -0800 (Pacific Standard Time) Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: <002001be6f1e$d20e3100$f4160218@c1004579-c.plstn1.sfba.home.com> Message-ID: > The conjugate gradient algorithm is probably about twenty lines or less of > matrix/vector statements in Python, assuming you have a preconditioner you > can express that way. So just code it up in Python. It will be fast enough, > all the hard work is in the dot products and matrix multiplies. CG is fine for some cases, but not all. What I miss in Python is the choices offered by e.g. Matlab's Optimization Toolbox: - Unconstrained Optimization - Quasi-Newton - Least Squares - Nonlinear Least Squares - Gauss-Newton - Levenberg Marquart - Constrained optimization - Sequential quadratic programming - Multiobjective optimization [taken from the TOC]. I suspect that if I knew enough about optimization I could code up the subset that I need, but I'm not sure I'd trust my own code to do this... --david From jac@lanl.gov Mon Mar 15 20:33:01 1999 From: jac@lanl.gov (James A. Crotinger) Date: Mon, 15 Mar 1999 13:33:01 -0700 Subject: [Matrix-SIG] Mailer insanity Message-ID: <199903152033.NAA11144@cic-mail.lanl.gov> Paul Dubois received a blank reply from me today. I don't know what happened - when I tried to read my mailbox this morning, my mailer went insane. I got it shut down, but now I notice that it lists a number of email messages in my matrix-sig mailbox as having been replied to. Again, no idea what happened, but I apologize for the crazy mailer. Take care, Jim ------------------------------------------------------------------------ James A. Crotinger Los Alamos National Lab email: jac@lanl.gov Technical Staff Member CIC-ACL, MS B287 phone: 505-665-6022 Pooma Team Los Alamos, NM 87545 fax: 505-665-4939 From janne@avocado.pc.helsinki.fi Mon Mar 15 21:09:22 1999 From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen) Date: 15 Mar 1999 23:09:22 +0200 Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: "Paul F. Dubois"'s message of "Mon, 15 Mar 1999 12:02:57 -0800" References: <002001be6f1e$d20e3100$f4160218@c1004579-c.plstn1.sfba.home.com> Message-ID: "Paul F. Dubois" writes: > The conjugate gradient algorithm is probably about twenty lines or > less of matrix/vector statements in Python, assuming you have a > preconditioner you can express that way. So just code it up in > Python. It will be fast enough, all the hard work is in the dot > products and matrix multiplies. I'm coding right now, just wanted to know if somebody has already done it. The hard work is often in the objective function, and from this viewpoing C is unnecessary unless the objective function is very fast. But a larger selection of optimization routines, either in Python or in C, would be nice (and probably emerges) sooner or later. -- Janne From janne@avocado.pc.helsinki.fi Mon Mar 15 21:25:16 1999 From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen) Date: 15 Mar 1999 23:25:16 +0200 Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: David Ascher's message of "Mon, 15 Mar 1999 12:10:07 -0800 (Pacific Standard Time)" References: Message-ID: David Ascher writes: > I suspect that if I knew enough about optimization I could code up > the subset that I need, but I'm not sure I'd trust my own code to do > this... Coding CG or some such with a simple line search is easy enough, but getting the line search robust may be more difficult. And robust stuff is hard to code also because it may "work" (although suboptimally) even with grave errors in the code. :) -- Janne From ryszard@moldyn.com Tue Mar 16 13:09:08 1999 From: ryszard@moldyn.com (Ryszard Czerminski) Date: Tue, 16 Mar 1999 08:09:08 -0500 (EST) Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: Message-ID: On 15 Mar 1999, Janne Sinkkonen wrote: > David Ascher writes: > > > I suspect that if I knew enough about optimization I could code up > > the subset that I need, but I'm not sure I'd trust my own code to do > > this... > > Coding CG or some such with a simple line search is easy enough, but > getting the line search robust may be more difficult. And robust stuff > is hard to code also because it may "work" (although suboptimally) even > with grave errors in the code. :) > > -- > Janne I have coded this up few month ago (based on NR). It is simple line search (without derivatives though). It seems to work OK. Ryszard def SIGN(a,b): if b < 0: return -abs(a) else : return abs(a) def SWAP(a,b): return b,a def mnbrak(ax,bx,cx,fa,fb,fc,func): # # brakets minimum of 1D function # i.e. returns a,b,c values such that fb < min(fa,fb) # GOLD = 1.618034; GLIMIT = 100.; TINY = 1.0e-20; dum = 0. fa = func(ax); fb = func(bx) if fb > fa: ax,bx = SWAP(ax,bx) fa,fb = SWAP(fa,fb) cx = bx + GOLD*(bx-ax) # first guess for c fc = func(cx) #print 'mnbrak> a,b,c,fa,fb,fc = ',ax,bx,cx,fa,fb,fc #iter = 0 while fb > fc: #iter = iter + 1 #print 'mnbrak> iter,a,b,c,fa,fb,fc = ',iter,ax,bx,cx,fa,fb,fc r = (bx-ax)*(fb-fc) q = (bx-cx)*(fb-fa) u = bx - ((bx-cx)*q - (bx-ax)*r) / (2*SIGN(max(abs(q-r),TINY),q-r)) ulim = bx + GLIMIT*(cx-bx) if (bx-u)*(u-cx) > 0: fu = func(u) if fu < fc: ax = bx; bx = u fa = fb; fb = fu return ax,bx,cx,fa,fb,fc elif fu > fb: cx = u; fc = fu return ax,bx,cx,fa,fb,fc u = cx + GOLD*(cx-bx) fu = func(u) elif (cx-u)*(u-ulim) > 0: fu = func(u) if fu < fc: bx = cx; cx = u; u = cx + GOLD*(cx-bx) fb = fc; fc = fu; fu = func(u) else: u = cx + GOLD*(cx-bx) fu = func(u) ax = bx; bx = cx; cx = u fa = fb; fb = fc; fc = fu return ax,bx,cx,fa,fb,fc def brent(ax,bx,cx,f,tol,maxiter): # # minimizes 1D function # ZEPS = 1.0e-10 CGOLD = 0.381966 a = min(ax,cx) b = max(ax,cx) x = w = v = bx fx = fw = fv = f(x) e = 0. iter = 0 while iter < maxiter: iter = iter + 1 xm = (a+b)/2 tol1 = tol*abs(x) + ZEPS tol2 = 2*tol1 if abs(x-xm) < (tol2-(b-a)/2): # test for convergence return x, fx if abs(e) > tol: r = (x-w)*(fx-fv) q = (x-v)*(fx-fw) p = (x-v)*q - (x-w)*r q = 2*(q-r) if q > 0: p = -p q = abs(q) etemp = e e = d if abs(p) >= abs(q*etemp/2) or p <= q*(a-x) or p >= q*(b-x): if x >= xm: e = a-x else : e = b-x d = CGOLD*e else: d = p/q u = x + d if u-a < tol2 or b-u < tol2: d = SIGN(tol1,xm-x) else: if x >= xm: e = a-x else : e = b-x d = CGOLD*e if abs(d) >= tol1: u = x + d else : u = x + SIGN(tol1,d) fu = f(u) if fu < fx: if u >= x : a = x else : b = x v = w; w = x; x = u fv = w; fw = fx; fx = fu else: if u < x : a = u else : b = u if fu <= fw or w == x: v = w; w = u; fv = fw; fw = fu elif fu <= fv or v == x or v == w: v = u; fv = fu print 'brent> error: iter, maxiter = ',iter, maxiter return x, fx From ryszard@moldyn.com Tue Mar 16 13:24:02 1999 From: ryszard@moldyn.com (Ryszard Czerminski) Date: Tue, 16 Mar 1999 08:24:02 -0500 (EST) Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: Message-ID: On Mon, 15 Mar 1999, David Ascher wrote: > CG is fine for some cases, but not all. What I miss in Python is the > choices offered by e.g. Matlab's Optimization Toolbox: > > - Unconstrained Optimization > - Quasi-Newton > - Least Squares > - Nonlinear Least Squares > - Gauss-Newton > - Levenberg Marquart > - Constrained optimization > - Sequential quadratic programming > - Multiobjective optimization > > [taken from the TOC]. > > I suspect that if I knew enough about optimization I could code up the > subset that I need, but I'm not sure I'd trust my own code to do this... > > --david If you could provide FRAMEWORK for this (i.e. place to deposit/distribute the code, etc.) and possibly some NumPy/Python specific help, I believe people who need optimization routines in their everyday work (like Janne, myself, others...) would populate it with useful code with time. Ryszard From ryszard@moldyn.com Tue Mar 16 13:27:08 1999 From: ryszard@moldyn.com (Ryszard Czerminski) Date: Tue, 16 Mar 1999 08:27:08 -0500 (EST) Subject: [Matrix-SIG] Nonlinear optimization routines anyone? In-Reply-To: Message-ID: On Mon, 15 Mar 1999, David Ascher wrote: > > does anybody has neatly packaged nonlinear optimization routines > > implemented in NumPy? I'd like to have either conjugate gradients or > > BFGS. Explicitly calculating the Hessian is out of question (because > > of the size and complexity of the problem). > > > > I thought translating part of the matlab codes of C. T. Kelley > > (http://www4.ncsu.edu/eos/users/c/ctkelley/www/matlab_darts.html) to > > Numerical Python unless something already implemented emerges. > > I've never found a routine to package, and I've looked a fair bit. There's > COOOL (http://timna.mines.edu/cwpcodes/coool/) which looked interesting, > but it seemed extremely non-portable (couldn't get it to compile on Win32, > requires expect, etc. etc.). > > --david As far as least squares problems are concernded at least one possible place is: C http://www.netlib.org/lawson-hanson/all C C This package of Fortran 77 and Fortran 90 code accompanies C the SIAM Publications printing of "Solving Least Squares C Problems," by C. Lawson and R. Hanson. The routines in the C package are filed as shown in the list below. The units C listed 1.-14. may be compiled and placed in a library. Routine C G2 is no longer used but is included for completeness. The C main program units 15.-18., 20.-21. require this library for C all external references. The item 19. is a data file read C by program unit PROG4.FOR. The main program 23., PROG7.F90, C requires the subprogram 22. or BVLS.F90. Ryszard From vanandel@atd.ucar.edu Wed Mar 17 18:29:24 1999 From: vanandel@atd.ucar.edu (Joe Van Andel) Date: Wed, 17 Mar 1999 11:29:24 -0700 Subject: [Matrix-SIG] GNU Scientific Library Message-ID: <199903171829.LAA08357@stout.atd.ucar.edu> Reply-To: vanandel@ucar.edu Cc: Fcc: -------- See http://sourceware.cygnus.com/gsl/ This isn't "production quality" yet, but I wanted to make sure you all saw this announcment, particularly since they state that they are interested in having it "wrapped". ------------------------- The GNU Scientific Library (GSL) is a collection of routines for numerical computing. The routines are written from scratch by the GSL team in ANSI C, and are meant to present a modern Applications Programming Interface (API) for C programmers, -> while allowing wrappers to be written for very high level languages. -< GSL is currently in developers release, for people who want to work on the library itself. When the library is complete and fully tested it will be announced for general use. --------------------------------------- Sounds like we might want to make this available for use with NumPy, if possible. Joe VanAndel Internet: vanandel@ucar.edu National Center for Web: http://www.atd.ucar.edu/~vanandel/home.html Atmospheric Research From alawhead@vcn.bc.ca Thu Mar 18 19:47:03 1999 From: alawhead@vcn.bc.ca (Alexander Lawhead) Date: Thu, 18 Mar 1999 11:47:03 -0800 (PST) Subject: [Matrix-SIG] algorithm for intelligent axis scaling? In-Reply-To: <199903171829.LAA08357@stout.atd.ucar.edu> Message-ID: This is not directly a NumPy question, but I figured that the best answer might come from this group. :) I'm looking for a simple algorithm for generating intelligent axis scalings for a graph. Basically what I need is the axis minimum, axis maximum, and step length between ticks given the minimum data value, maximum data value, and suggested number of ticks. By intelligent, I mean reasonably intuitive rounding for the axis labels! This should be easy, but I'm afraid I've been spoiled by automagical plotting in countless canned graphics packages. Any help would be appreciated. Thanks, Alexander From rburnham@cri-inc.com Thu Mar 18 20:48:58 1999 From: rburnham@cri-inc.com (Roger Burnham) Date: Thu, 18 Mar 1999 12:48:58 -800 Subject: [Matrix-SIG] algorithm for intelligent axis scaling? In-Reply-To: References: <199903171829.LAA08357@stout.atd.ucar.edu> Message-ID: <199903182049.OAA15281@smtp2.gte.net> On 18 Mar 99, at 11:47, Alexander Lawhead mused: > This is not directly a NumPy question, but I figured that the best answer > might come from this group. :) I'm looking for a simple algorithm for > generating intelligent axis scalings for a graph. Basically what I need is > the axis minimum, axis maximum, and step length between ticks given the > minimum data value, maximum data value, and suggested number of ticks. > By intelligent, I mean reasonably intuitive rounding for the axis labels! > > This should be easy, but I'm afraid I've been spoiled by automagical > plotting in countless canned graphics packages. Any help would be > appreciated. > Just today, I downloaded Richard Jones' PILGraph 0.1 package that was recently announced on c.l.p. Available at http://redback.spyda.net/~richard/PILGraph-0.1a1.tar.gz It has just such an algo. Cheers, Roger Burnham Cambridge Research & Instrumentation rburnham@cri-inc.com http://www.cri-inc.com/ http://starship.python.net/crew/roger/ PGP Key: http://www.nai.com/default_pgp.asp PGP Fingerprint: 5372 729A 9557 5F36 177F 084A 6C64 BE27 0BC4 CF2D From da@ski.org Thu Mar 18 21:13:35 1999 From: da@ski.org (David Ascher) Date: Thu, 18 Mar 1999 13:13:35 -0800 (Pacific Standard Time) Subject: [Matrix-SIG] algorithm for intelligent axis scaling? In-Reply-To: Message-ID: > This is not directly a NumPy question, but I figured that the best answer > might come from this group. :) I'm looking for a simple algorithm for > generating intelligent axis scalings for a graph. Basically what I need is > the axis minimum, axis maximum, and step length between ticks given the > minimum data value, maximum data value, and suggested number of ticks. > By intelligent, I mean reasonably intuitive rounding for the axis labels! I've found it surprisingly difficult. The routine I use in Snow (http://starship.python.net:9673/crew/da/Snow) is: def format(v): if int(v) == v: return `int(v)` else: return `v` def myticks(self, Min, Max, log, fitMin=1, fitMax=1): r = [] ; l = [] # what's the tick seperation: if min is .2, then it's .1 # if it's .002, then it's 0.001. if log: startscale = floor(log10(Min)) endscale = floor(log10(Max)) candidates = [] for x in arange(startscale, endscale+1): candidates = candidates + (arange(1,10)*power(10,x)).tolist() valid = nonzero(logical_and(greater(candidates, Min), less(candidates, Max))) doable = candidates[valid[0]-1:valid[-1]+2] for v in doable: if v/power(10,floor(log10(v))) == 1.0: r.append((v,0)) elif v/power(10,floor(log10(v))) == 5.0: r.append((v,1)) else: r.append((v,2)) if (v/power(10, floor(log10(v))) in (1.0, 5.0)): l.append((v, None, format(v))) # enable following for ticks at startat and endat if 0 and (doable[0], doable[-1]): l.append((v, None, format(v))) return doable[0], r, l, doable[-1] else: # linear case d = abs(Max - Min) epsilon = 1e-10 s = power(10,floor(log10(d))-1) if fitMin: startat = Min - (Min % (s * 10)) else: startat = Min overmax = Max + (s * 10) if fitMax: endat = overmax - (overmax % (s * 10)) else: endat = Max if d / (10*s) > 5.2: # magic! # small tickmarks at unit factors s = s * 10 # we need to upshift the scale maj_mod = 10 # major tick marks at factors of 10 int_mod = 2 # intermediate tick marks at factors of 2 else: maj_mod = 10 # major tick marks at factors of 10 int_mod = 5 # minors at factors of 5 # smalls at factors of 1 r = [] ; l = [] nummajors = 0 # for debugging numints = 0 for x in arange(startat, endat+epsilon, s): if (abs((x / s) % maj_mod) < epsilon): x = round(x, 4) r.append((x,0)) l.append((x, None, format(x))) nummajors = nummajors + 1 elif (abs((x / s) % int_mod) < epsilon): x = round(x, 4) r.append((x,1)) l.append((x, None, format(x))) numints = numints + 1 else: x = round(x, 4) r.append((x,2)) return startat, r, l, endat It's a hack, but it's neater than the code in gist. =) It's also highly undocumented. Playing with it however should make it relatively clear what the return values are. --david From dubois1@llnl.gov Fri Mar 19 01:23:39 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Thu, 18 Mar 1999 17:23:39 -0800 Subject: [Matrix-SIG] LLNL Distribution #10 Message-ID: <001301be71a7$1edf9980$f4160218@c1004579-c.plstn1.sfba.home.com> LLNL Distribution #10, like the truth, is out there: ftp-icf.llnl.gov/pub/python Get LLNLDistribution.tgz for the source. Get NumPy.exe for the Windows 95/98/NT installer. Gist Fixed mesh3d.py to handle tetrahedra, and to allow data of either type 'd' or type 'b'. Fixed slice3.py so that data type will not be changed. Numerous changes in slice2 and _slice2_part in gistCmodule to support data values of either type 'b' or type 'd'. Bug fixes in gistCmodule: make sure result arrays are returned as Py_None in case a slice is empty; clear the OWN_DATA pointer in a PyArrayObject when the pointer has been given to a newly created object. The color card specification for a 3D object will now overrule the specification (if any) for the Graph containing the object. The default style sheet for a Graph3d will be "nobox.gs". A Graph3D style specification will be overruled by "z_nobox.gs" if a color bar is requested, otherwise uncritically accepted. Numpy Fixed a bug in the NumPy exponentiation routine which was causing a SEGV when an array of Python longs was an argument. Fixed memory leak in array to list conversion. (Jonah Lee) Fixed segv if you attempt to delete an array element. (Warren Focke) Fixed bug in convolve in multiarray.c (Travis Oliphant) Added missing #!... to makethis.py in Numerical (Pearu Peterson, who also reported Gist probs). Fixed documentation of clip. (Andrew Sterian) Changed array so that it will accept a typecode of None. This allows simplifications to indices, asarray, and UserArray. (Keith Junius) Performance improvement for FFT.py (Konrad Hinsen) PDB Added import_array to PyPDB module init routine. From gathmann@scar.utoronto.ca Fri Mar 19 12:57:41 1999 From: gathmann@scar.utoronto.ca (Oliver Gathmann) Date: Fri, 19 Mar 1999 07:57:41 -0500 (EST) Subject: [Matrix-SIG] algorithm for intelligent axis scaling? In-Reply-To: Message-ID: On Thu, 18 Mar 1999, Alexander Lawhead wrote: > This is not directly a NumPy question, but I figured that the best answer > might come from this group. :) I'm looking for a simple algorithm for > generating intelligent axis scalings for a graph. Basically what I need is > the axis minimum, axis maximum, and step length between ticks given the > minimum data value, maximum data value, and suggested number of ticks. > By intelligent, I mean reasonably intuitive rounding for the axis labels! > I'm afraid this might not be needed anymore, but I guess it's good to have as many different opinions on how to solve a particular problem as possible...: from Numeric import * def _calcMinMax(self, x, p = 0.05): ''' Compute initial min and max axis scaling points. Approach: a) with buffer: reserve a fraction p of the total span of an axis as buffer and round to next order of magnitude b) strict (p==0): just round to the next order of magnitude Special cases: x_min==x_max : assign symmetric interval or [0,1], if zero. ''' if len(x)>0: # not an empty array passed x_max,x_min = maximum.reduce(x),minimum.reduce(x) if x_min <> x_max: # esp. not both x_min,x_max equal to zero span = x_max - x_min buffer = p * span if x_min-buffer > 0: # both (x_min-buffer),(x_max+buffer) > 0 x_min = round(x_min - buffer, -(floor(log10(buffer) - 1))) x_max = round(x_max + buffer, -(ceil(log10(buffer) - 1))) elif x_max+buffer < 0: # both (x_min-buffer),(x_max+buffer) < 0 x_min = round(x_min - buffer, -(ceil(log10(buffer) - 1))) x_max = round(x_max + buffer, -(floor(log10(buffer) - 1))) else: # (x_min-buffer /= 0) try: x_min = round(x_min - buffer, -(ceil(log10(buffer) - 1))) except OverflowError: # buffer == 0 x_min = 0 try: x_max = round(x_max + buffer, -(ceil(log10(buffer) - 1))) except OverflowError: # buffer == 0 x_max = 0 else: if x_min <> 0: x_min = x_min - x_min/2.0 x_max = x_max + x_max/2.0 else: x_min = 0 x_max = 1 else: x_min = 0 x_max = 1 return x_min,x_max Regards, Oliver F. Oliver Gathmann (gathmann@scar.utoronto.ca) Surface and Groundwater Ecology Research Group University of Toronto phone: (416) - 287 7420 ; fax: (416) - 287 7423 web: http://www.scar.utoronto.ca/~gathmann From miller5@uiuc.edu Fri Mar 19 14:52:12 1999 From: miller5@uiuc.edu (Mike Miller) Date: 19 Mar 1999 08:52:12 -0600 Subject: [Matrix-SIG] algorithm for intelligent axis scaling? References: Message-ID: <76pv65352b.fsf@zero.npl.uiuc.edu> Note: I've CC'd this to plot-sig as well as matrix-sig. Might be some people there who'll be interested in this discussion. >>>>> On matrix-sig, Alexander Lawhead writes: > This is not directly a NumPy question, but I figured that > the best answer might come from this group. :) I'm looking > for a simple algorithm for generating intelligent axis > scalings for a graph. Basically what I need is the axis > minimum, axis maximum, and step length between ticks given > the minimum data value, maximum data value, and suggested > number of ticks. By intelligent, I mean reasonably > intuitive rounding for the axis labels! > This should be easy, but I'm afraid I've been spoiled by > automagical plotting in countless canned graphics > packages. Any help would be appreciated. Here's how I do it in some DISLIN utilities. Scales calculates what I think are ok values for linear or log10 scales and sets up dislin's axes with dislin.graf. If I'm plotting xy pairs with uncertainties dx and dy, Scales is called like this: Scales( min(x), max(x), min(y-dy), max(y+dy) ) On the aesthetic scale, this version leans more to serviceable than beautiful - I'm glad to see the other methods that have been posted. Thanks, Mike -- Michael A. Miller miller5@uiuc.edu Department of Physics, University of Illinois, Urbana-Champaign # --- excerpts from DISLIN PlutUtils.py --- # Default number of divisions on each axis: Ndiv = 10 def Scales(x1, x2, y1, y2, xlog='lin', ylog='lin'): if xlog == 'log': xmin, xmax, xlowTick, xStep = log10Scale(x1,x2) dislin.scale('LOG','X') dislin.labels('LOG','X') else: xmin, xmax, xlowTick, xStep, xdig = linearScale(x1,x2) dislin.digits(xdig, 'x') if ylog == 'log': ymin, ymax, ylowTick, yStep = log10Scale(y1,y2) dislin.scale('LOG','Y') dislin.labels('LOG','Y') else: ymin, ymax, ylowTick, yStep, ydig = linearScale(y1,y2) dislin.digits(ydig, 'y') dislin.graf( xmin, xmax, xlowTick, xStep, ymin, ymax, ylowTick, yStep ) def linearScale(a1,a2): width = float(a2) - float(a1) step = width/float(Ndiv) istep = pow(10.0,int(cmath.log10(step).real)) Step = math.ceil(step/istep)*istep lowTick = math.floor(a1/Step)*Step if a1 == 0.0: amin = a1 else: amin = a1 - Step/2.0 amax = a2 + Step/2.0 if lowTick < min: lowTick = lowTick + Step if amax-amin > 10.0: adig = 0 else: adig = - math.floor(math.log10(amax-amin)) + 1 return amin, amax, lowTick, Step, adig def log10Scale(a1, a2): amax = math.ceil(math.log10(a2)) if a1 <= 0.0: amin = amax - 3 else: amin = math.floor(math.log10(a1)) return amin, amax, amin, 1.0 From dubois1@llnl.gov Fri Mar 19 15:32:52 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 19 Mar 1999 07:32:52 -0800 Subject: [Matrix-SIG] LLNL Distribution #10 Message-ID: <002d01be721d$c2e729a0$f4160218@c1004579-c.plstn1.sfba.home.com> They keep changing the darn thing and evidently I guessed wrong on how to do it. I have 1.5.2 and it worked fine. I will try to get it fixed ASAP. In the meantime, either stick to distribution 9 or upgrade to 1.5.2. Thanks for letting me know. -----Original Message----- From: Les Schaffer To: Paul F. Dubois Date: Thursday, March 18, 1999 8:38 PM Subject: [Matrix-SIG] LLNL Distribution #10 >>>>>> ">" == Paul F Dubois writes: > > >> LLNL Distribution #10, like the truth, is out there: > >> ftp://ftp-icf.llnl.gov/pub/python > >the install failed for me on window 98 with a 'cant find python >1.5'. > >i have 1.5.1 on the system..... > >les > From HYoon@exchange.ml.com Fri Mar 19 15:39:26 1999 From: HYoon@exchange.ml.com (Yoon, Hoon (CICG - NY Program Trading)) Date: Fri, 19 Mar 1999 10:39:26 -0500 Subject: [Matrix-SIG] LLNL Distribution #10 Message-ID: Hi, I just downloaded Numpy.exe. It reports "Sorry Python 1.5 does not seem to be on your system," but I have 1.5.1 available. I am not sure what it is looking for. I also reported a bug about NaN == NaN on NT platform where as on Unix NaN != NaN. Has this been added to the bug list or fixed? If not, then can you pls add to the bug list? Thank you much for the package, ************************************************************** S. Hoon Yoon (Quant) Merrill Lynch Equity Trading yelled@yahoo.com hoon@bigfoot.com(w) "Miracle is always only few standard deviations away, but so is catastrophe." * Expressed opinions are often my own, but NOT my employer's. "I feel like a fugitive from the law of averages." Mauldin ************************************************************** > -----Original Message----- > From: Paul F. Dubois > Sent: Thursday, March 18, 1999 8:24 PM > To: matrix-sig@python.org > Subject: [Matrix-SIG] LLNL Distribution #10 > > LLNL Distribution #10, like the truth, is out there: > > ftp-icf.llnl.gov/pub/python > > Get LLNLDistribution.tgz for the source. > Get NumPy.exe for the Windows 95/98/NT installer. > > Gist > Fixed mesh3d.py to handle tetrahedra, and to allow data of either type 'd' > or type 'b'. > Fixed slice3.py so that data type will not be changed. > Numerous changes in slice2 and _slice2_part in gistCmodule to support data > values of either type 'b' or type 'd'. > Bug fixes in gistCmodule: make sure result arrays are returned as Py_None > in > case a slice is empty; clear the OWN_DATA pointer in a PyArrayObject when > the pointer has been given to a newly created object. > The color card specification for a 3D object will now overrule the > specification (if any) for the Graph containing the object. > The default style sheet for a Graph3d will be "nobox.gs". A Graph3D style > specification will be overruled by "z_nobox.gs" if a color bar is > requested, > otherwise uncritically accepted. > > Numpy > Fixed a bug in the NumPy exponentiation routine which was causing a SEGV > when an array of Python longs was an argument. > Fixed memory leak in array to list conversion. (Jonah Lee) > Fixed segv if you attempt to delete an array element. (Warren Focke) > Fixed bug in convolve in multiarray.c (Travis Oliphant) > Added missing #!... to makethis.py in Numerical (Pearu Peterson, who also > reported Gist probs). > Fixed documentation of clip. (Andrew Sterian) > Changed array so that it will accept a typecode of None. This allows > simplifications to indices, asarray, and UserArray. (Keith Junius) > Performance improvement for FFT.py (Konrad Hinsen) > > PDB > Added import_array to PyPDB module init routine. > > > > > > > > _______________________________________________ > Matrix-SIG maillist - Matrix-SIG@python.org > http://www.python.org/mailman/listinfo/matrix-sig From hinsen@cnrs-orleans.fr Fri Mar 19 15:47:24 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 19 Mar 1999 16:47:24 +0100 Subject: [Matrix-SIG] algorithm for intelligent axis scaling? In-Reply-To: (message from Oliver Gathmann on Fri, 19 Mar 1999 07:57:41 -0500 (EST)) References: Message-ID: <199903191547.QAA24208@dirac.cnrs-orleans.fr> > I'm afraid this might not be needed anymore, but I guess it's good to have > as many different opinions on how to solve a particular problem as > possible...: Right, so I'll add mine as well (from my Tk plotting widget). You call it with the lower and upper limits of what you want to see, and it returns a list of tick marks (value/string pairs): import Numeric def ticks(lower, upper): ideal = (upper-lower)/7. log = Numeric.log10(ideal) power = Numeric.floor(log) fraction = log-power factor = 1. error = fraction for f, lf in multiples: e = Numeric.fabs(fraction-lf) if e < error: error = e factor = f grid = factor * 10.**power if power > 3 or power < -3: format = '%+7.0e' elif power >= 0: digits = max(1, int(power)) format = '%' + `digits`+'.0f' else: digits = -int(power) format = '%'+`digits+2`+'.'+`digits`+'f' ticks = [] t = -grid*Numeric.floor(-lower/grid) while t <= upper and len(ticks) < 200: ticks.append(t, format % (t,)) t = t + grid return ticks multiples = [(2., Numeric.log10(2.)), (5., Numeric.log10(5.))] To decide about the limits to feed to ticks(), I use the following function. It is called with the smallest and largest value in the dataset and returns updated values for the limits according to one of two strategies (minimal plot box or "nicely rounded"): def axisInterval(spec, lower, upper): if spec == 'minimal': if lower == upper: return lower-0.5, upper+0.5 else: return lower, upper if spec == 'automatic': range = upper-lower if range == 0.: return lower-0.5, upper+0.5 log = Numeric.log10(range) power = Numeric.floor(log) fraction = log-power if fraction <= 0.05: power = power-1 grid = 10.**power lower = lower - lower % grid mod = upper % grid if mod != 0: upper = upper - mod + grid return lower, upper if type(spec) == type(()): lower, upper = spec if lower <= upper: return lower, upper else: return upper, lower raise ValueError, str(spec) + ': illegal axis specification' -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From dubois1@llnl.gov Fri Mar 19 19:32:22 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 19 Mar 1999 11:32:22 -0800 Subject: [Matrix-SIG] LLNL Distribution #10 Message-ID: <001001be723f$36306540$f4160218@c1004579-c.plstn1.sfba.home.com> Yes. fast_umath was identical to umath. I forgot to mention this deletion in the notes. -----Original Message----- From: Les Schaffer To: Paul F. Dubois Date: Friday, March 19, 1999 9:59 AM Subject: Re: [Matrix-SIG] LLNL Distribution #10 >also, i noticed the unix tar.gz file is missing the fast_math >stuff. is that by design? > >les >-- >____ Les Schaffer ___| --->> Engineering R&D <<--- >Theoretical & Applied Mechanics | Designspring, Inc. >Center for Radiophysics & Space Research | http://www.designspring.com/ >Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com > From zow@llnl.gov Fri Mar 19 20:09:08 1999 From: zow@llnl.gov (Zow Terry Brugger) Date: Fri, 19 Mar 1999 12:09:08 -0800 Subject: [Matrix-SIG] Parallel matrix solvers Message-ID: <199903192009.MAA04327@pensive.llnl.gov> Good day, Our group is looking at using Python for large matrix solves, however it has come to my attention that the current NumPy implementation is serial. Has anyone done any work on using MPI and/or threads to parallelize huge matrix operations? Any leads on such a beast would be appreciated. I signed up for the list but I'm still waiting for the confirmation mail, so please reply to me directly. Thanks! "Zow" Terry Brugger zow@llnl.gov From amullhau@zen-pharaohs.com Fri Mar 19 20:36:17 1999 From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt) Date: Fri, 19 Mar 1999 15:36:17 -0500 Subject: [Matrix-SIG] Parallel matrix solvers Message-ID: <017301be7248$252d7360$ef11d3c6@amullhau.ix.netcom.com> -----Original Message----- From: Zow Terry Brugger To: matrix-sig@python.org Date: Friday, March 19, 1999 3:25 PM Subject: [Matrix-SIG] Parallel matrix solvers >Good day, > Our group is looking at using Python for large matrix solves, however it has >come to my attention that the current NumPy implementation is serial. Has >anyone done any work on using MPI and/or threads to parallelize huge matrix >operations? That's a pretty specific kind of parallelization which isn't really that good, but a lot of work has been done on libraries to provide this sort of parallelism in libraries for numerical linear algebra (see www.netlib.org). You could extend Numerical Python to call these functions. The reason that message passing is a particularly lackluster form of parallelism is mainly that it forces the programmer to write all the communication code, and this has normally resulted in code which is good for a few architectures but not so good for the most available - symmetric multiprocessors. Later, Andrew Mullhaupt From godzilla@netmeg.net (Les Schaffer) Fri Mar 19 21:02:43 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Fri, 19 Mar 1999 16:02:43 -0500 (EST) Subject: [Matrix-SIG] LLNL Distribution #10 In-Reply-To: <001001be723f$36306540$f4160218@c1004579-c.plstn1.sfba.home.com> References: <001001be723f$36306540$f4160218@c1004579-c.plstn1.sfba.home.com> Message-ID: <14066.47987.961354.100086@gargle.gargle.HOWL> > Yes. fast_umath was identical to umath. I forgot to mention this > deletion in the notes. okay. the tooth is, i never really understood what umath and fast_umath was (were). unsafe math? sounds like ....... on to Gist: on the latest release, i just compiled gist with -O2 -g -mpentium, and got a segfault in the 9th test in gisttest3d.py: (gustav)/usr/local/src/LLNLDistribution10/Graphics/Gist3D/Demo/: python gisttest3d.py env variable PYGRAPH is not set, so assuming Gist graphics. ...this can be overruled by the 'graphics' keyword argument to Graph2d or Graph3d. Opening graphics window 0. Type in any string to continue; ^C to return to prompt. [repeated 8 more times] Segmentation fault (core dumped) the problem occurs in the s3 = sslice(): # pseudo-colored plane slice, then cut it in half and save # only the front half s3 = sslice (m1, pyz, opt_3d = ["wm", "f4"]) s3 = sslice (s3, pxy, nslices = 1, opt_3d = ["wm", "f4"]) here's the core dump in gdb: (gustav)/usr/local/src/LLNLDistribution10/Graphics/Gist3D/Demo/: gdb python core GNU gdb 4.17.0.8 with Linux support [snip] This GDB was configured as "i586-pc-linux-gnu"... Core was generated by `python gisttest3d.py'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libreadline.so.2...done. [etc] Reading symbols from /usr/lib/python1.5/site-packages/graphics/arrayfnsmodule.so...done. #0 0x4041a933 in slice2 (self=0x0, args=0x811acf0) at ./Gist/Src/gistCmodule.c:5394 5394 valuecc = (Uchar *) (valuec->data); (gdb) print valuec $1 = (ArrayObject *) 0x0 (gdb) info args self = (PyObject *) 0x0 args = (PyObject *) 0x0 (gdb) bt #0 0x4041a933 in slice2 (self=0x0, args=0x811acf0) at ./Gist/Src/gistCmodule.c:5394 #1 0x8057242 in call_builtin () #2 0x8057158 in PyEval_CallObjectWithKeywords () #3 0x80564d5 in eval_code2 () #4 0x80563ce in eval_code2 () #5 0x80563ce in eval_code2 () #6 0x80563ce in eval_code2 () #7 0x8057f66 in PyEval_EvalCode () #8 0x8069a7f in run_node () #9 0x8069a44 in run_err_node () #10 0x80697d0 in PyRun_File () #11 0x8068ba7 in PyRun_SimpleFile () #12 0x80695da in PyRun_AnyFile () #13 0x8052fbe in Py_Main () #14 0x8052a98 in main () -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From dubois1@llnl.gov Fri Mar 19 21:06:46 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 19 Mar 1999 13:06:46 -0800 Subject: [Matrix-SIG] version number is in error Message-ID: <000301be724c$65f21f00$f4160218@c1004579-c.plstn1.sfba.home.com> In the Numerical distribution #10 I forgot to change the "version numbers" at the head of Numerical/Lib/Numeric.py. These version numbers were put in by a cabal interested in such things, which I generally oppose because they are quite often wrong, especially when it is me who has to keep them right. From Oliphant.Travis@mayo.edu Fri Mar 19 23:38:25 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 19 Mar 1999 17:38:25 -0600 (EST) Subject: [Matrix-SIG] RPMS of latest Release 10: Message-ID: I've wrapped up the latest release as RPM files: Binaries for i386 Linux compiled with gcc available at http://oliphant.netpedia.net/RPMS (get all python-numpy-* files) Source RPMS so you can compile your own rpm on any platform. (RPM is available for many platforms besides Linux --- see www.rpm.org) http://oliphant.netpedia.net/SRPMS I've included the LLNL provided documentation for gist in this package and the postscript files in the /pub/python directory at LLNL as well. (I've noticed the segfault bug mentioned by Les in gistCmodule also. It very likely has something to do with support for both 'd' and 'b' types just added.) --Travis From Oliphant.Travis@mayo.edu Tue Mar 23 05:06:31 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 22 Mar 1999 23:06:31 -0600 (EST) Subject: [Matrix-SIG] Updated version of Signaltools now includes numpyio Message-ID: I've updated my website a bit at http://oliphant.netpedia.net Links to much of the LLNLDistribution10 as RPMS are now more obvous: I've updated the sigtools module to include some Python-written functions. The package also now contains (a memory-leak-fixed) numpyio module for reading and writing data from arbitrary binary files directly into NumPy arrays. Although there is still a module called sigtools, the module to load is now signaltools.py . This file imports all the C-code functions but also defines other routines: convolveND : the C-code routine is renamed correlateND to be more consistent with signal processing usage and this routine performs an actual index reversal on the smallest input so it is a true convolution. medfiltND: makes use of the more general order_filterND to define a more commonly used median filter. wienerND: performs wiener filtering over an N-D array. Notice, the licensing has changed a bit on signaltools. Basically, you can't include it in binary-form only in a commerical product without letting me know (From the responses I've received I doubt anyone is using it much at all, though). If you are using any package I release, I'd love to hear about it. Especially, since I can gauge what kind of impact making big changes will have. Thanks, Travis From dubois1@llnl.gov Tue Mar 23 23:51:41 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Tue, 23 Mar 1999 15:51:41 -0800 Subject: [Matrix-SIG] Numerical Python document Message-ID: <000401be7588$19bd7000$f4160218@c1004579-c.plstn1.sfba.home.com> The first draft of the Numerical Python document is now available. You can get it at http://xfiles.llnl.gov. Follow the link to the Python documentation and get the PDF file and the accompanying NumTut tutorial file by clicking on the links. I thought it was better to get it out with a few imperfections than to sit on it. Later this week I hope to get these files moved over to the X-Division ftp site, ftp-icf.llnl.gov, where the sources and Windows installer for NumPy reside. However, this FTP site was recently moved to a new computer and I have not yet received the instructions from X-Division as to how to get my files onto it, so I was unable to do that yet. Since xfiles is not a high-capacity server, please be patient; this early announcement is being set to the matrix-sig only and I would think things will behave better if nobody makes a general announcement until I get things moved to the better server. Thanks and kudos to David Ascher, who wrote the document, Travis Oliphant and Konrad Hinsen, who contributed, and Jim Hugunin, who wrote most of the code and original documentation. You can send corrections to me at dubois1@llnl.gov. From godzilla@netmeg.net (Les Schaffer) Wed Mar 24 16:11:00 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Wed, 24 Mar 1999 11:11:00 -0500 (EST) Subject: [Matrix-SIG] fromfunction broke in LLNL10?? Message-ID: <14073.3732.227931.400233@gargle.gargle.HOWL> I just was using some old code with a Numeric.fromfunction, and now its broke, after upgrade to LLNL10 from 9. did something change in Numeric.py? Traceback (innermost last): File "/usr/lib/python1.5/lib-tk/Tkinter.py", line 761, in __call__ return apply(self.func, args) File "testImage.py", line 58, in update ra = fromfunction(self.func, (101, 101)) File "/usr/lib/python1.5/site-packages/Numerical/Numeric.py", line 156, in fromfunction return apply(function, tuple(indices(dimensions))) File "/usr/lib/python1.5/site-packages/Numerical/Numeric.py", line 149, in indices tmp = ones(dimensions, typecode) File "/usr/lib/python1.5/site-packages/Numerical/Numeric.py", line 306, in ones return zeros(shape, typecode)+array(1, typecode) TypeError: argument 2: expected string, None found this code worked flawlessly before 10: =============================== #!/usr/bin/python from Tkinter import * from ImageTk import PhotoImage import Image from Numeric import * def normalize(a): a = array(a, copy=1) a = (a - min(a.flat))/(max(a.flat) - min(a.flat)) return a def ArrayToImage(a, height=200, scale=1): if len(a.shape) == 2: a = (normalize(a)*254).astype('b') i = Image.new("L", (a.shape[1], a.shape[0]), color=255) i.fromstring(a.tostring()) elif len(a.shape) == 1: a = 1-normalize(a) a = array(a*height).astype('i') i = Image.new("L", (a.shape[0], height), color=255) id = ImageDraw.ImageDraw(i) id.setink(0) def map_xandy_to_xy(x, y): import _imaging return _imaging.path(ravel(transpose(array((x, y)))).tolist()) id.line(xy=map_xandy_to_xy(arange(a.shape[0]), a)) if scale != 1: i = i.resize((i.size[0]*scale, i.size[1]*scale)) return i def ImageToArray(i): a = fromstring(i.tostring(), 'b') a.shape = i.im.size[1], i.im.size[0] return a class MatrixDisplay: def __init__(self, master): frame = Frame(master) frame.pack() self.button = Button(frame, text="QUIT", fg="red", command=frame.quit) self.button.pack(side=TOP) self.test_B = Button(frame, text="Update", command=self.update) self.test_B.pack(side=TOP) self.img = Image.new("L",(100,100)) self.photo = PhotoImage(image=self.img) self.canvas = Canvas(frame, width=100, height=100) self.canvas.create_image(0, 0, anchor=NW, image=self.photo) self.canvas.pack() def update(self): ra = fromfunction(self.func, (101, 101)) self.img = ArrayToImage(ra) #self.photo = PhotoImage(self.img) self.photo.paste(self.img) self.canvas.create_image(0, 0, anchor=NW, image=self.photo) self.canvas.update() def func(self, i,j): return cos( 4*2*3.1415926*i / 100. ) * cos( 4*2*3.1415926*(j / 100. )) root = Tk() app = MatrixDisplay(root) root.mainloop() ============================== -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From godzilla@netmeg.net (Les Schaffer) Wed Mar 24 17:49:50 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Wed, 24 Mar 1999 12:49:50 -0500 (EST) Subject: [Matrix-SIG] fromfunction broke in LLNL10?? In-Reply-To: <36F91F55.B551BDD3@llnl.gov> References: <14073.3732.227931.400233@gargle.gargle.HOWL> <36F91B27.A4E0D8E5@llnl.gov> <14073.7809.915052.979815@gargle.gargle.HOWL> <36F91F55.B551BDD3@llnl.gov> Message-ID: <14073.9662.281139.366363@gargle.gargle.HOWL> > Hm. I suspect it was a change to "indices". If you don't want to > wait for me to get a chance to look at it, you might just get the > indices from version 9 and paste it in and see if that fixes it. The > change was a simplification that was supposed to be possible now; I > got it from a user. yup... i assume this wont break something else. Les -------------- start patch ----------------- *** Numeric.py~ Thu Mar 18 13:52:23 1999 --- Numeric.py Wed Mar 24 12:46:40 1999 *************** *** 146,152 **** return reshape(a, new_shape) def indices(dimensions, typecode=None): ! tmp = ones(dimensions, typecode) lst = [] for i in range(len(dimensions)): lst.append( add.accumulate(tmp, i, )-1 ) --- 146,155 ---- return reshape(a, new_shape) def indices(dimensions, typecode=None): ! if typecode == None: ! tmp = ones(dimensions) ! else: ! tmp = ones(dimensions, typecode) lst = [] for i in range(len(dimensions)): lst.append( add.accumulate(tmp, i, )-1 ) ---------------- end patch -------------- -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From dubois1@llnl.gov Wed Mar 24 23:58:28 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Wed, 24 Mar 1999 15:58:28 -0800 Subject: [Matrix-SIG] Numerical Python Documentation and Tutorial Message-ID: <000e01be7652$36690280$37860cc0@almanac.llnl.gov> Numerical Python Documentation and Tutorial Paul F. Dubois - March 24th 1999, 16:00 PST The Numerical Python document and a tutorial package that accompanies it are now available. See: http://xfiles.llnl.gov, or ftp://ftp-icf.llnl.gov/pub/python/README.html. Authors: Ascher, Dubois, Hinsen, Hugunin, Oliphant License: free redistribution as explained in legal notice in document. ----------- comp.lang.python.announce (moderated) ---------- Article Submission Address: python-announce@python.org Python Language Home Page: http://www.python.org/ Python Quick Help Index: http://www.python.org/Help.html ------------------------------------------------------------ From pearu@ioc.ee Fri Mar 26 10:41:54 1999 From: pearu@ioc.ee (Pearu Peterson) Date: Fri, 26 Mar 1999 12:41:54 +0200 (EET) Subject: [Matrix-SIG] Numerical Python Documentation and Tutorial In-Reply-To: <000e01be7652$36690280$37860cc0@almanac.llnl.gov> Message-ID: Is it possible to make The Numerical Python document available in other formats like HTML, TEX, the source? Thanks, Pearu Pearu Peterson , MSc, Researcher Department of Mechanics and Applied Mathematics http://koer.ioc.ee/~pearu/ Institute of Cybernetics at Tallinn Technical University Phone: (+372) 6204168 Akadeemia Rd. 21, 12618 Tallinn ESTONIA Fax: (+372) 6204151 *** the nonvalidity of rigorous causality is necessary and not just consistently possible (Heisenberg, 1925) *** On Wed, 24 Mar 1999, Paul F. Dubois wrote: > Numerical Python Documentation and Tutorial > Paul F. Dubois - March 24th 1999, 16:00 PST > > The Numerical Python document and a tutorial package that accompanies it are > now available. See: > > http://xfiles.llnl.gov, or > ftp://ftp-icf.llnl.gov/pub/python/README.html. > > Authors: Ascher, Dubois, Hinsen, Hugunin, Oliphant > License: free redistribution as explained in legal notice in document. > From dubois1@llnl.gov Fri Mar 26 15:48:21 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 26 Mar 1999 07:48:21 -0800 Subject: [Matrix-SIG] Numerical Python Documentation and Tutorial Message-ID: <000601be77a0$13a05560$f4160218@c1004579-c.plstn1.sfba.home.com> The html version is now on http://xfiles.llnl.gov/NumDoc4.html. This file is automatically produced; please don't write about any HTML formatting errors. If you want to see the document the way it was intended, please use the Acrobat version. -----Original Message----- From: Pearu Peterson To: Paul F. Dubois Cc: matrix-sig@python.org Date: Friday, March 26, 1999 3:13 AM Subject: Re: [Matrix-SIG] Numerical Python Documentation and Tutorial > >Is it possible to make The Numerical Python document available in other >formats like HTML, TEX, the source? > >Thanks, > Pearu > >Pearu Peterson , MSc, Researcher >Department of Mechanics and Applied Mathematics http://koer.ioc.ee/~pearu/ >Institute of Cybernetics at Tallinn Technical University Phone: (+372) 6204168 >Akadeemia Rd. 21, 12618 Tallinn ESTONIA Fax: (+372) 6204151 >*** the nonvalidity of rigorous causality is necessary > and not just consistently possible (Heisenberg, 1925) *** > >On Wed, 24 Mar 1999, Paul F. Dubois wrote: > >> Numerical Python Documentation and Tutorial >> Paul F. Dubois - March 24th 1999, 16:00 PST >> >> The Numerical Python document and a tutorial package that accompanies it are >> now available. See: >> >> http://xfiles.llnl.gov, or >> ftp://ftp-icf.llnl.gov/pub/python/README.html. >> >> Authors: Ascher, Dubois, Hinsen, Hugunin, Oliphant >> License: free redistribution as explained in legal notice in document. >> > > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From Oliphant.Travis@mayo.edu Fri Mar 26 19:04:22 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 26 Mar 1999 13:04:22 -0600 (EST) Subject: [Matrix-SIG] Re: NumPy questions In-Reply-To: <199903261648.KAA06612@v1.wustl.edu> Message-ID: > > I've been working thru the newly released Numerical Python document > and subscribed to the matrix SIG. But, I'm having many problems. > Where would be the appropriate place to direct these questions > in the future so I can stop bugging you? The matrix-sig is a good place to ask questions. Then you can get answers from many people besides me (multiple insights). I don't mind answering questions when I can help, though. Email matrix-sig@python.org to have your mail go to the list. I'm posting this repsonse to that list, too. Archives of the list can be viewed at http://www.python.org/pipermail/matrix-sig/ If your question is more general (not NumPy specific) you can get a wider audience by posting to comp.lang.python as you did before. > > Also, just so you know, I'm considering using NumPy to rewrite > a bunch of stuff for cortical surface extraction & flattening. > .... I think NumPy/Python could save me a > lot of time, but only if I can figure it out! > > I have an array defined in python: > > >>> b > array([[ 7, 8, 9, 9, 9, 9, 9, 11], > [14, 15, 16, 15, 15, 17, 19, 13], > [ 7, 9, 12, 13, 14, 13, 12, 10], > [ 9, 12, 15, 13, 11, 9, 7, 8], > [10, 10, 10, 9, 9, 7, 6, 6], > [ 7, 9, 11, 11, 12, 12, 13, 13], > [13, 11, 10, 9, 9, 9, 9, 11], > [14, 16, 18, 16, 14, 11, 9, 10], > [12, 11, 11, 10, 9, 10, 11, 15], > [20, 16, 13, 13, 13, 18, 23, 26], > [30, 21, 12, 9, 7, 8, 10, 11], > [13, 13, 14, 12, 10, 9, 8, 8], > [ 9, 11, 13, 13, 14, 16, 18, 17], > [16, 13, 11, 11, 11, 11, 12, 13], > [15, 15, 16, 12, 8, 7, 7, 8]],'b') > > I try to perform a searchsorted operation on it: > > >>> searchsorted (b, arange (0.0,255,1.0)) > Traceback (innermost last): > File "", line 1, in ? > ValueError: Object too deap for desired array > You've noticed that sometimes the error messages have typos. These should be fixed. > What does this error message mean? I searched all the comp.lang.python > postings and the www.python.org website. Why is my array too "deap"?? Searching for an explanation of error messages is often not fruitful in Numerical. The only real explanation of all of the messages is in the source code. The error messages themselves are supposed to be somewhat helpful in determining the problem. In this case the error message type is ValueError. This means that an incorrect value was passed to the function. The explanation is that the object is too "deap" (deep). Depth of an object refers to how many dimensions it has. Here the searchsorted function works only with rank-1 arrays (i.e. 1 dimensional arrays) so it can't be used with a matrix like this. > It's an 15x8 matrix which brings up another question: normally, > the first number is the number of columns. In python, it's the > number of rows...?? Are matrices sorted in row-major order? column-major? > I don't know what you mean by normally. NumPy defines a generic array object that can have any number of dimensions (actually it's restricted to no more than 40 dimensions at the moment). Row-major and column-major order are specific for matrices only. With that said, look at b.shape: >>> print b.shape (15, 8) So b is an array of rank-2 that has 15 elements along the first dimension (number of rows) and 8 elements along the second dimension (number of columns). This fits well with interpretation that b is a 15x8 matrix. I don't now what you mean by "sorted in row-major order...". If you are talking about how the data is stored in memory, then if the array is contiguous (not the result of a slicing operation) it is stored with the last dimension varying the fastest. If it is not contiguous then ravel will give you the elements as a rank-1 (1-D) array with the last dimension varying the fastest and the first dimension varying the slowest: >>> ravel(b) array([ 7, 8, 9, 9, 9, 9, 9, 11, 14, 15, 16, 15, 15, 17, 19, 13, 7, 9, 12, 13, 14, 13, 12, 10, 9, 12, 15, 13, 11, 9, 7, 8, 10, 10, 10, 9, 9, 7, 6, 6, 7, 9, 11, 11, 12, 12, 13, 13, 13, 11, 10, 9, 9, 9, 9, 11, 14, 16, 18, 16, 14, 11, 9, 10, 12, 11, 11, 10, 9, 10, 11, 15, 20, 16, 13, 13, 13, 18, 23, 26, 30, 21, 12, 9, 7, 8, 10, 11, 13, 13, 14, 12, 10, 9, 8, 8, 9, 11, 13, 13, 14, 16, 18, 17, 16, 13, 11, 11, 11, 11, 12, 13, 15, 15, 16, 12, 8, 7, 7, 8],'b') > I try a few of the other functions: > > >>> sort (b,0) > Traceback (innermost last): > File "", line 1, in ? > File "/usr/local/lib/python1.5/NumPy/Numeric.py", line 73, in sort > if axis != -1: a = swapaxes(a, axis, -1) > File "/usr/local/lib/python1.5/NumPy/Numeric.py", line 49, in swapaxes > n = len(shape(a)) > File "/usr/local/lib/python1.5/NumPy/Numeric.py", line 278, in shape > return asarray(a).shape > File "/usr/local/lib/python1.5/NumPy/Numeric.py", line 274, in asarray > return array(a, typecode, copy=0) > TypeError: argument 2: expected string, None found > I'm not sure what is happening here as when I enter sort(b,0) I get the array b sorted along the first (0th) axis. Using matrix terminology it is sorted down each column. To sort along each row then use sort(b,1) { = sort(b,-1) = sort(b) }. Again, this does not happen to me, has b changed somehow? >>> b array([[ 7, 8, 9, 9, 9, 9, 9, 11], [14, 15, 16, 15, 15, 17, 19, 13], [ 7, 9, 12, 13, 14, 13, 12, 10], [ 9, 12, 15, 13, 11, 9, 7, 8], [10, 10, 10, 9, 9, 7, 6, 6], [ 7, 9, 11, 11, 12, 12, 13, 13], [13, 11, 10, 9, 9, 9, 9, 11], [14, 16, 18, 16, 14, 11, 9, 10], [12, 11, 11, 10, 9, 10, 11, 15], [20, 16, 13, 13, 13, 18, 23, 26], [30, 21, 12, 9, 7, 8, 10, 11], [13, 13, 14, 12, 10, 9, 8, 8], [ 9, 11, 13, 13, 14, 16, 18, 17], [16, 13, 11, 11, 11, 11, 12, 13], [15, 15, 16, 12, 8, 7, 7, 8]],'b') >>> sort(b,0) array([[ 7, 8, 9, 9, 7, 7, 6, 6], [ 7, 9, 10, 9, 8, 7, 7, 8], [ 7, 9, 10, 9, 9, 8, 7, 8], [ 9, 10, 11, 9, 9, 9, 8, 8], [ 9, 11, 11, 10, 9, 9, 9, 10], [10, 11, 11, 11, 9, 9, 9, 10], [12, 11, 12, 11, 10, 9, 9, 11], [13, 12, 12, 12, 11, 10, 10, 11], [13, 13, 13, 12, 11, 11, 11, 11], [14, 13, 13, 13, 12, 11, 12, 13], [14, 15, 14, 13, 13, 12, 12, 13], [15, 15, 15, 13, 14, 13, 13, 13], [16, 16, 16, 13, 14, 16, 18, 15], [20, 16, 16, 15, 14, 17, 19, 17], [30, 21, 18, 16, 15, 18, 23, 26]],'b') > Well, I don't know what's wrong with this, as b is a 2-D matrix > and I'm sorting using the 0th axis... This should work just fine. Try >>> shape(b) >>> asarray(b) > Does it need to be square?? No > > These are a few of the examples of the problems I'm having. > I hope we can help you resolve them. - Travis From Oliphant.Travis@mayo.edu Fri Mar 26 19:40:47 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 26 Mar 1999 13:40:47 -0600 (EST) Subject: [Matrix-SIG] Re: NumPy addendum (Error in Documentation) In-Reply-To: <199903261702.LAA06660@v1.wustl.edu> Message-ID: > Travis, > > I got: > > histogram ((ravel(b)).astype('d'), arange (0.0,255.0,1.0)) > > to work. I see that the input matrix to search sorted needed > to have rank 1 (my fault), but why does it have to be type 'd'? > I've found the problem finally. It's a mistake in the documentation: searchsorted should be called with the bin_boundaries as the first argument and the data to be sorted as the second argument. This is done in the example but when histogram is defined the naming convention used gets confusing. So, as it stands you should really say searchsorted( arange(0.0,255.0,1.0),ravel(b)) or in fact you could use searchsorted( arange(0,255,1,'b'),ravel(b)) to have comparisons between unsigned integers. and the documentation should be changed. I still think there is an implementation bug, though, and you should not get the error you got (even if you did call the function incorrectly --- being led astray by the documentation). The error you say is due to the fact that the underlying function has to make comparisons between numbers. These numbers must be the same type for the comparsion to work. Therefore, the two input arrays have to have the same type. As you had it before your two arrays were of different types. b was an unsigned char and arange(0.0,255.0,1.0) is an array of double type. So, one has to be typecast to the other. The way it's implemented now, the first array determines the type of the second. And so the code tries to convert your arange into unsigned chars which it complains about because it would lose information. As is done in other routines, this should really convert both arrays to the "highest" type, so you wouldn't have to convert it yourself. Best, Travis From Oliphant.Travis@mayo.edu Fri Mar 26 20:01:13 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 26 Mar 1999 14:01:13 -0600 (EST) Subject: [Matrix-SIG] Ignore much of previous posting (not a documentation Error). Message-ID: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. ---1333709111-1252024020-922478473=:6335 Content-Type: TEXT/PLAIN; charset=US-ASCII Err... Sorry about falsely waving the error flag in the last post. I misunderstood the histogram function in the documentation. It is written correctly. I do still think the implementation bug in mutliarraymodule should be fixed and so here is a patch to make it possible to call searchsorted so that either argument will be coerced appropriately to make comparisons. -- Travis ---1333709111-1252024020-922478473=:6335 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=patch1 Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Context diff file to patch against multiarraymodule.c Content-Disposition: attachment; filename=patch1 ZGlmZiAtY3IgTExOTERpc3RyaWJ1dGlvbjEwLm9yaWcvTnVtZXJpY2FsL1Ny Yy9tdWx0aWFycmF5bW9kdWxlLmMgTExOTERpc3RyaWJ1dGlvbjEwL051bWVy aWNhbC9TcmMvbXVsdGlhcnJheW1vZHVsZS5jDQoqKiogTExOTERpc3RyaWJ1 dGlvbjEwLm9yaWcvTnVtZXJpY2FsL1NyYy9tdWx0aWFycmF5bW9kdWxlLmMJ VGh1IE1hciAxOCAxMjo1MjoyNiAxOTk5DQotLS0gTExOTERpc3RyaWJ1dGlv bjEwL051bWVyaWNhbC9TcmMvbXVsdGlhcnJheW1vZHVsZS5jCUZyaSBNYXIg MjYgMTM6NTk6MTYgMTk5OQ0KKioqKioqKioqKioqKioqDQoqKiogNDkyLDUw NCAqKioqDQogIAljaGFyICppcDsNCiAgCWxvbmcgKnJwOw0KICAJQ29tcGFy ZUZ1bmN0aW9uIGNvbXBhcmVfZnVuYzsNCiEgCQ0KICAJcmV0ID0gTlVMTDsN CiEgCWFwMSA9IChQeUFycmF5T2JqZWN0ICopUHlBcnJheV9Db250aWd1b3Vz RnJvbU9iamVjdChvcDEsIFB5QXJyYXlfTk9UWVBFLCAxLA0KISAJCTEpOw0K ICAJaWYgKGFwMSA9PSBOVUxMKSByZXR1cm4gTlVMTDsNCiEgCWFwMiA9IChQ eUFycmF5T2JqZWN0ICopUHlBcnJheV9Db250aWd1b3VzRnJvbU9iamVjdChv cDIsDQohIAkJYXAxLT5kZXNjci0+dHlwZV9udW0sMCwwKTsNCiAgCWlmIChh cDIgPT0gTlVMTCkgZ290byBmYWlsOw0KICAJDQogIAlyZXQgPSAoUHlBcnJh eU9iamVjdCAqKVB5QXJyYXlfRnJvbURpbXMoYXAyLT5uZCwgYXAyLT5kaW1l bnNpb25zLA0KLS0tIDQ5Miw1MDUgLS0tLQ0KICAJY2hhciAqaXA7DQogIAls b25nICpycDsNCiAgCUNvbXBhcmVGdW5jdGlvbiBjb21wYXJlX2Z1bmM7DQoh IAlpbnQgdHlwZW51bSA9IDA7DQohIA0KISAgICAgICAgIHR5cGVudW0gPSBQ eUFycmF5X09iamVjdFR5cGUob3AxLCAwKQ0KISAgICAgICAgIHR5cGVudW0g PSBQeUFycmF5X09iamVjdFR5cGUob3AyLCB0eXBlbnVtKQkNCiAgCXJldCA9 IE5VTEw7DQohIAlhcDEgPSAoUHlBcnJheU9iamVjdCAqKVB5QXJyYXlfQ29u dGlndW91c0Zyb21PYmplY3Qob3AxLCB0eXBlbnVtLCAxLCAxKTsNCiAgCWlm IChhcDEgPT0gTlVMTCkgcmV0dXJuIE5VTEw7DQohIAlhcDIgPSAoUHlBcnJh eU9iamVjdCAqKVB5QXJyYXlfQ29udGlndW91c0Zyb21PYmplY3Qob3AyLCB0 eXBlbnVtLCAwLCAwKTsNCiAgCWlmIChhcDIgPT0gTlVMTCkgZ290byBmYWls Ow0KICAJDQogIAlyZXQgPSAoUHlBcnJheU9iamVjdCAqKVB5QXJyYXlfRnJv bURpbXMoYXAyLT5uZCwgYXAyLT5kaW1lbnNpb25zLA0K ---1333709111-1252024020-922478473=:6335-- From Ted.Horst@wdr.com Mon Mar 29 18:06:14 1999 From: Ted.Horst@wdr.com (Ted Horst) Date: Mon, 29 Mar 99 12:06:14 -0600 Subject: [Matrix-SIG] fromfunction broken in LLNL Distribution #10 ? Message-ID: <9903291806.AA01308@ch1d2833nwk> When I try fromfunction, I get the following exception: >>> Numeric.fromfunction(lambda x:x, 10) Traceback (innermost last): File "", line 1, in ? File "/home/horst/tmp/python/NumPy/Numeric.py", line 156, in fromfunction return apply(function, tuple(indices(dimensions))) File "/home/horst/tmp/python/NumPy/Numeric.py", line 149, in indices tmp = ones(dimensions, typecode) File "/home/horst/tmp/python/NumPy/Numeric.py", line 306, in ones return zeros(shape, typecode)+array(1, typecode) TypeError: argument 2: expected string, None found In distribution #6 there was a check in indices for typecode == None, this will fix the problem, but a better solution is probably to put the checking for Py_None in array_zeros like it is in array_array. Also the doc string for zeros incorrectly states that the default type is 'd'. It is 'l'. Same for fromstring. Here is a patch: *** multiarraymodule.c.orig Mon Mar 29 11:39:45 1999 --- multiarraymodule.c Mon Mar 29 11:54:32 1999 *************** *** 880,889 **** return ret; } ! static char doc_zeros[] = "zeros(d1,...,dn,typecode='d') will return a new array of shape (d1,...,dn) and type typecode with all it's entries initialized to zero."; static PyObject *array_zeros(PyObject *ignored, PyObject *args) { ! PyObject *op, *sequence; PyArrayObject *ret; char type_char='l'; char *type = &type_char, *dptr; --- 880,889 ---- return ret; } ! static char doc_zeros[] = "zeros(d1,...,dn,typecode='l') will return a new array of shape (d1,...,dn) and type typecode with all it's entries initialized to zero."; static PyObject *array_zeros(PyObject *ignored, PyObject *args) { ! PyObject *op, *sequence, *tpo=Py_None; PyArrayObject *ret; char type_char='l'; char *type = &type_char, *dptr; *************** *** 890,897 **** int i, nd, n, dimensions[MAX_DIMS]; char all_zero[16] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}; ! if (PyArg_ParseTuple(args, "O|s", &sequence, &type) == NULL) return NULL; if ((nd=PySequence_Length(sequence)) == -1) { PyErr_Clear(); if (!(op = PyNumber_Int(sequence))) return NULL; --- 890,905 ---- int i, nd, n, dimensions[MAX_DIMS]; char all_zero[16] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}; ! if (PyArg_ParseTuple(args, "O|O", &sequence, &tpo) == NULL) return NULL; + if (tpo != Py_None) { + type = PyString_AsString(tpo); + if (!type) + return NULL; + if (!*type) + type = &type_char; + } + if ((nd=PySequence_Length(sequence)) == -1) { PyErr_Clear(); if (!(op = PyNumber_Int(sequence))) return NULL; *************** *** 924,930 **** return (PyObject *)ret; } ! static char doc_fromString[] = "fromString(string, count=None, typecode='d') returns a new 1d array initialized from the raw binary data in string. If count is not equal to None, the new array will have count elements, otherwise it's size is determined by the siuze of string."; static PyObject *array_fromString(PyObject *ignored, PyObject *args) { PyArrayObject *ret; --- 932,938 ---- return (PyObject *)ret; } ! static char doc_fromString[] = "fromstring(string, count=None, typecode='l') returns a new 1d array initialized from the raw binary data in string. If count is not equal to None, the new array will have count elements, otherwise it's size is determined by the size of string."; static PyObject *array_fromString(PyObject *ignored, PyObject *args) { PyArrayObject *ret; From dubois1@llnl.gov Mon Mar 29 19:24:54 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Mon, 29 Mar 1999 11:24:54 -0800 Subject: [Matrix-SIG] fromfunction broken in LLNL Distribution #10 ? Message-ID: <000601be7a19$d3245fa0$f4160218@c1004579-c.plstn1.sfba.home.com> Thanks for the patch. I got a fix from a user for release 9 that broke this inadvertently; I guess I need to broaden the test suite at some point because it didn't test fromfunciton. Another fix people can make to work around this is to pick up the previous version of function "indices" in Numerical.py. -----Original Message----- From: Ted Horst To: matrix-sig@python.org Date: Monday, March 29, 1999 10:13 AM Subject: [Matrix-SIG] fromfunction broken in LLNL Distribution #10 ? > >When I try fromfunction, I get the following exception: > >>>> Numeric.fromfunction(lambda x:x, 10) >Traceback (innermost last): > File "", line 1, in ? > File "/home/horst/tmp/python/NumPy/Numeric.py", line 156, in fromfunction > return apply(function, tuple(indices(dimensions))) > File "/home/horst/tmp/python/NumPy/Numeric.py", line 149, in indices > tmp = ones(dimensions, typecode) > File "/home/horst/tmp/python/NumPy/Numeric.py", line 306, in ones > return zeros(shape, typecode)+array(1, typecode) >TypeError: argument 2: expected string, None found > >In distribution #6 there was a check in indices for typecode == None, this will fix the problem, but a better solution is probably to put the checking for Py_None in array_zeros like it is in array_array. > >Also the doc string for zeros incorrectly states that the default type is 'd'. It is 'l'. Same for fromstring. > >Here is a patch: > >*** multiarraymodule.c.orig Mon Mar 29 11:39:45 1999 >--- multiarraymodule.c Mon Mar 29 11:54:32 1999 >*************** >*** 880,889 **** > return ret; > } > >! static char doc_zeros[] = "zeros(d1,...,dn,typecode='d') will return a new array of shape (d1,...,dn) and type typecode with all it's entries initialized to zero."; > > static PyObject *array_zeros(PyObject *ignored, PyObject *args) { >! PyObject *op, *sequence; > PyArrayObject *ret; > char type_char='l'; > char *type = &type_char, *dptr; >--- 880,889 ---- > return ret; > } > >! static char doc_zeros[] = "zeros(d1,...,dn,typecode='l') will return a new array of shape (d1,...,dn) and type typecode with all it's entries initialized to zero."; > > static PyObject *array_zeros(PyObject *ignored, PyObject *args) { >! PyObject *op, *sequence, *tpo=Py_None; > PyArrayObject *ret; > char type_char='l'; > char *type = &type_char, *dptr; >*************** >*** 890,897 **** > int i, nd, n, dimensions[MAX_DIMS]; > char all_zero[16] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}; > >! if (PyArg_ParseTuple(args, "O|s", &sequence, &type) == NULL) return NULL; > > if ((nd=PySequence_Length(sequence)) == -1) { > PyErr_Clear(); > if (!(op = PyNumber_Int(sequence))) return NULL; >--- 890,905 ---- > int i, nd, n, dimensions[MAX_DIMS]; > char all_zero[16] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}; > >! if (PyArg_ParseTuple(args, "O|O", &sequence, &tpo) == NULL) return NULL; > >+ if (tpo != Py_None) { >+ type = PyString_AsString(tpo); >+ if (!type) >+ return NULL; >+ if (!*type) >+ type = &type_char; >+ } >+ > if ((nd=PySequence_Length(sequence)) == -1) { > PyErr_Clear(); > if (!(op = PyNumber_Int(sequence))) return NULL; >*************** >*** 924,930 **** > return (PyObject *)ret; > } > >! static char doc_fromString[] = "fromString(string, count=None, typecode='d') returns a new 1d array initialized from the raw binary data in string. If count is not equal to None, the new array will have count elements, otherwise it's size is determined by the siuze of string."; > > static PyObject *array_fromString(PyObject *ignored, PyObject *args) { > PyArrayObject *ret; >--- 932,938 ---- > return (PyObject *)ret; > } > >! static char doc_fromString[] = "fromstring(string, count=None, typecode='l') returns a new 1d array initialized from the raw binary data in string. If count is not equal to None, the new array will have count elements, otherwise it's size is determined by the size of string."; > > static PyObject *array_fromString(PyObject *ignored, PyObject *args) { > PyArrayObject *ret; > > > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From dubois1@llnl.gov Mon Mar 29 23:21:11 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Mon, 29 Mar 1999 15:21:11 -0800 Subject: [Matrix-SIG] Re: error in installing Numerical on windows Message-ID: <002601be7a3a$d5729260$f4160218@c1004579-c.plstn1.sfba.home.com> I made an error in putting together the Windows installer. It works with Python 1.5.2 but not 1.5.1. You can grab the "release 9" file for Numerical Python or you can upgrade to 1.5.2 or you can wait for the next Numerical. I'm sure there is some equivalent where you edit the registry but if I understood that properly I wouldn't have made the mistake in the first place. The next release of Numerical will not need to fiddle with the registry to get the path right, thus ridding of this problem forever. I hope to get it out in the next week or so. -----Original Message----- From: Lienhart, Rainer To: support@icf.llnl.gov Date: Monday, March 29, 1999 2:58 PM >Hi, >today I tried to install NumPy.exe under NT4.0 SP4 on a dual Xeon-CPU >machine. Hereto, I grabbed python 1.5.1 and win32all from www.python.org, >installed it and rebooted the machine. When I called NumPy.exe and got the >following error message: > Sorry, > the installation of the Numerical can be performed. > This is because Python version 1.5 does not appear to be installed >on this computer > You can find this version of Python at www.python.org >I had no problem to install RNG.exe. Do you know any solution to this >problem. >Thanks, > Rainer > >--------------------------------------------------------------------------- - >------ >Rainer Lienhart. Ph.D. >Sr. Researcher >Micro-computer Research Labs, Intel Corporation, >2200 Mission College Blvd., Santa Clara, CA 95052. >Phone: (408) 765-3450 Fax: (408) 653-8511 >Email: Rainer.Lienhart@intel.com > > > From dubois1@llnl.gov Tue Mar 30 20:29:23 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Tue, 30 Mar 1999 12:29:23 -0800 Subject: [Matrix-SIG] 11beta.tgz Message-ID: <000301be7aeb$ffaa32c0$f4160218@c1004579-c.plstn1.sfba.home.com> Dear Numerical Pythoners, Please get ftp://ftp-icf.llnl.gov/pub/python/11beta.tgz and try it out. It is a beta release of LLNL Distribution 11. The reason it is a beta release is that it has a new install procedure and a new attributes feature that could very well break, leak memory, or otherwise irritate the heck out of somebody. (Then again, maybe it works, and I like it.) This version should be fine on Windows too but you'll need a compiler since the tar file doesn't have the compiled binaries in it. The new install procedure (if it works) requires Python 1.5. Please let me know if it works for you, or not. Thanks, Paul Dubois From cgw@fnal.gov Tue Mar 30 21:56:33 1999 From: cgw@fnal.gov (Charles G Waldman) Date: Tue, 30 Mar 1999 15:56:33 -0600 (CST) Subject: [Matrix-SIG] 11beta.tgz In-Reply-To: <000301be7aeb$ffaa32c0$f4160218@c1004579-c.plstn1.sfba.home.com> References: <000301be7aeb$ffaa32c0$f4160218@c1004579-c.plstn1.sfba.home.com> Message-ID: <14081.18577.696416.997485@buffalo.fnal.gov> Paul F. Dubois writes: > > Please let me know if it works for you, or not. > Works like a champ for me, with: gcc version egcs-2.91.66 19990314 (egcs-1.1.2 release) Linux 2.2.5 #19 Tue Mar 30 10:59:21 CST 1999 i686 unknown Python 1.5.2b2 (#12, Mar 26 1999, 15:00:52) [GCC egcs-2.91.66 19990314 (egcs-1.1.2 on linux2 From frohne@alaska.net Wed Mar 31 00:22:54 1999 From: frohne@alaska.net (Ivan Frohne) Date: Tue, 30 Mar 1999 15:22:54 -0900 Subject: [Matrix-SIG] Additional Ufuncts Message-ID: <000001be7b0c$9ea6ad20$469270d1@vladimir> Although unmentioned in either the Numeric Reference or the new (and very readable) tutorial "Numerical Python," I found the following ufuncs: ceil(x) floor(x) fabs(x) fmod(x, y) left_shift(i, count) right_shift(i, count) hypot(x) in the Numeric module. The ones I tried worked as expected on Numeric arrays. --Ivan Frohne From Oliphant.Travis@mayo.edu Wed Mar 31 03:59:11 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Tue, 30 Mar 1999 21:59:11 -0600 (EST) Subject: [Matrix-SIG] We should rename convolve in multiarraymodule Message-ID: While I've mentioned this privately to some members of the list, I thought I might suggest it to all users of NumPy to see the reaction. I'm suggesting that "convolve" be renamed to "correlate." as this is what the function actually does. It is only the same as a true convolution if one of the inputs is symmetric. It is quite easy to obtain a convolution from a correlation function as it just involves reversing the axis on one of the sequences. In linear system theory, convolution has a very special place and anyone coming from that background to use Numerical Python will be quite surprised and a bit annoyed that the convolve function is actually computing a correlation. The difference between convolution and correlation is subtle but important for some problems, and I would like to make sure that we don't lose potential users because of a mild oversight in the naming of one of the functions. In the signaltools package that I've put together, there is a function called convolveND (true convolution) and a function called correlateND (that can replace the current "convolve" in NumPy). Thanks for indulging, Travis From frank@ned.dem.csiro.au Wed Mar 31 04:39:16 1999 From: frank@ned.dem.csiro.au (Frank Horowitz) Date: Wed, 31 Mar 1999 12:39:16 +0800 Subject: [Matrix-SIG] We should rename convolve in multiarraymodule In-Reply-To: Message-ID: At 11:59 AM +0800 31/3/99, Travis Oliphant wrote: >While I've mentioned this privately to some members of the list, I thought >I might suggest it to all users of NumPy to see the reaction. > >I'm suggesting that "convolve" be renamed to "correlate." as this is what >the function actually does. It is only the same as a true convolution if >one of the inputs is symmetric. It is quite easy to obtain a convolution >from a correlation function as it just involves reversing the axis on one >of the sequences. Might I suggest an alternative? How about making convolve do both things, triggered by an argument? I, for one, expect to find a convolution routine in a suite that purports to do signal processing, and might not think to look for such a routine under the "correlate" moniker. If backwards compatibility is truly a goal (I have my doubts it's such a huge problem in scripting environment such as Python), make the argument default to correlate. That way, people who actually want convolve() to convolve instead of correlate have a one-liner to insert: def convolve(...): return Numeric.convolve(...,conv='conv') for instance... I guess I'm mostly uncomfortable with naming the only routine to perform such an operation "correlate" instead of "convolve". I think "convolve" is more widely recognizable and used. If the current implementation is broken, fix it. Frank -- Frank Horowitz frank@ned.dem.csiro.au Australian Geodynamics Cooperative Research Centre, and CSIRO-Exploration & Mining, PO Box 437, Nedlands, WA 6009, AUSTRALIA Direct: +61 8 9284 8431; FAX: +61 8 9389 1906; Reception: +61 8 9389 8421 From r.hooft@euromail.net Wed Mar 31 07:07:14 1999 From: r.hooft@euromail.net (Rob Hooft) Date: Wed, 31 Mar 1999 09:07:14 +0200 (MZT) Subject: [Matrix-SIG] We should rename convolve in multiarraymodule In-Reply-To: References: Message-ID: <14081.51618.646220.633358@octopus.chem.uu.nl> >>>>> "FH" == Frank Horowitz writes: FH> If backwards compatibility is truly a goal (I have my doubts it's FH> such a huge problem in scripting environment such as Python), I'd like to hear you say that again if you're working on a 35k line python project involving calls to Numeric, PIL, Tkinter, Pmw and some other packages.... Python may be a scripting language but in some places it doesn't pay off to rewrite the code (or one just can't afford the effort). I just grepped through my code: not using convolve (yet)... Regards, -- ===== R.Hooft@EuroMail.net http://www.xs4all.nl/~hooft/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= From hinsen@cnrs-orleans.fr Wed Mar 31 13:39:50 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Wed, 31 Mar 1999 15:39:50 +0200 Subject: [Matrix-SIG] We should rename convolve in multiarraymodule In-Reply-To: (message from Travis Oliphant on Tue, 30 Mar 1999 21:59:11 -0600 (EST)) References: Message-ID: <199903311339.PAA21186@dirac.cnrs-orleans.fr> > I'm suggesting that "convolve" be renamed to "correlate." as this is what > the function actually does. It is only the same as a true convolution if I didn't check, because I never used this, but I agree that the names should correspond to the actual function. I'd prefer to have both "correlate" and "convolve", where of course one would be implemented in terms of the other with an added reverse operation. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From godzilla@netmeg.net (Les Schaffer) Wed Mar 31 14:26:12 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Wed, 31 Mar 1999 09:26:12 -0500 (EST) Subject: [Matrix-SIG] We should rename convolve in multiarraymodule In-Reply-To: References: Message-ID: <14082.12420.791560.287953@gargle.gargle.HOWL> > While I've mentioned this privately to some members of the list, I > thought I might suggest it to all users of NumPy to see the > reaction. yes, absolutely please correct the name. its way too easy to write a perl script to change instances of convolve to correlate in python codes, even 35k lines. btw, the latest pdf of the docs for NumPy has no entry for convolve in it -- its the null set actually, the title is there but the info is empty. (Page 52). les schaffer From ransom@cfa.harvard.edu Wed Mar 31 14:59:01 1999 From: ransom@cfa.harvard.edu (Scott M. Ransom) Date: Wed, 31 Mar 1999 08:59:01 -0600 Subject: [Matrix-SIG] We should rename convolve in multiarraymodule References: <14082.12420.791560.287953@gargle.gargle.HOWL> Message-ID: <37023835.D98FC3B@cfa.harvard.edu> Les Schaffer wrote: > > While I've mentioned this privately to some members of the list, I > > thought I might suggest it to all users of NumPy to see the > > reaction. > > yes, absolutely please correct the name. its way too easy to write a > perl script to change instances of convolve to correlate in python > codes, even 35k lines. I agree. And here is the Perl script: perl -p -i -e 's/convolve\(/correlate\(/o' *.py Cheers, Scott -- Scott M. Ransom Phone: (580) 536-7215 Address: 703 SW Chaucer Cir. email: ransom@cfa.harvard.edu Lawton, OK 73505 PGP Fingerprint: D2 0E D0 10 CD 95 06 DA EF 78 FE 2B CB 3A D3 53 From Oliphant.Travis@mayo.edu Wed Mar 31 15:23:28 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Wed, 31 Mar 1999 09:23:28 -0600 (EST) Subject: [Matrix-SIG] We should rename convolve in multiarraymodule In-Reply-To: <000b01be7b37$6ffe3120$f4160218@c1004579-c.plstn1.sfba.home.com> Message-ID: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. ---1333709111-467216601-922893808=:24988 Content-Type: TEXT/PLAIN; charset=US-ASCII > FWIW I agree completely. I've taken the liberty to create a patch against the 11beta source to fix this convolve glitch. The patch renames the "convolve" in the C-code to cross_correlate and defines a convolve in Numeric.py that calls cross_correlate with the elements of the smallest argument reversed. This should not break any code since convolve is still defined (now correctly) unless that code relied on the "broken" behavior of convolve. -- Travis Help with patch: If you have the patch utility, this patch can be applied by changing into the top-level directory of the distribution (i.e. LLNLDistribution11beta) and typing patch -p1 < full_path_name_to_corr_patch e.g. if corr_patch is in the directory where 11beta.tgz sits then tar zxvf 11beta.tgz cd LLNLDistribution11beta patch -p1 <../corr_patch ---1333709111-467216601-922893808=:24988 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=corr_patch Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Use diff -p1 I have applied your patch. I do anything if it is easy (:->. Thanks for the help. Since neither cross_correlate or convolve has a manual entry, would you be willing to send me an entry on each? -----Original Message----- From: Travis Oliphant To: Paul F. Dubois Cc: matrix-sig@python.org Date: Wednesday, March 31, 1999 7:28 AM Subject: Re: [Matrix-SIG] We should rename convolve in multiarraymodule >> FWIW I agree completely. > >I've taken the liberty to create a patch against the 11beta source to fix >this convolve glitch. The patch renames the "convolve" in the C-code to >cross_correlate and defines a convolve in Numeric.py that calls >cross_correlate with the elements of the smallest argument reversed. > >This should not break any code since convolve is still defined (now >correctly) unless that code relied on the "broken" behavior of convolve. > > >-- Travis > > > >Help with patch: > > >If you have the patch utility, this patch can be applied by changing into >the top-level directory of the distribution (i.e. LLNLDistribution11beta) >and typing > >patch -p1 < full_path_name_to_corr_patch > >e.g. if corr_patch is in the directory where 11beta.tgz sits then > >tar zxvf 11beta.tgz >cd LLNLDistribution11beta >patch -p1 <../corr_patch > > From godzilla@netmeg.net (Les Schaffer) Wed Mar 31 16:29:08 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Wed, 31 Mar 1999 11:29:08 -0500 (EST) Subject: [Matrix-SIG] We should rename convolve in multiarraymodule In-Reply-To: <37023835.D98FC3B@cfa.harvard.edu> References: <14082.12420.791560.287953@gargle.gargle.HOWL> <37023835.D98FC3B@cfa.harvard.edu> Message-ID: <14082.19796.617146.777055@gargle.gargle.HOWL> Scott Ransom writes: > I agree. And here is the Perl script: > perl -p -i -e 's/convolve\(/correlate\(/o' *.py i'd make that perl -p -i.orig -e 's/convolve/correlate/g' *.py so that you get backups of any files that are changed... or, i'll see your one-liner and raise you 28. usage (*nix): subst convolve correlate `find . -name "*.py"` subst will keep original copies of any file its changed, and also provide a listing of exactly what changes it did make (diff -c) to stdout. if it makes changes to file.py, it copies the permissions of the original to the updated version. the first and second arguments also take some subset of perl regexps if you have something more tricky. les schaffer =============== subst ==================== #!/usr/bin/perl my $PATTERN = $ARGV[0]; my $SUBPAT = $ARGV[1]; shift; shift; foreach $file ( @ARGV) { $NEWFILE = $file.".tmp"; $OLDFILE = $file.".orig"; open(FILE, $file); open(OUT, ">$NEWFILE"); @stats = stat $file; $mode = $stats[2]; while ( ) { $_ =~ s/\Q$PATTERN/\Q$SUBPAT/g; printf OUT "%s", $_; } close(FILE); close(OUT); @diff = `diff -c $file $NEWFILE `; printf ("%s", "@diff"); if ( $#diff < 0 ) { printf "No changes in %s ...\n", $file; unlink $NEWFILE; } else { rename $file, $OLDFILE ; rename $NEWFILE, $file; chmod $mode, $file; } } From gathmann@scar.utoronto.ca Wed Mar 31 17:10:35 1999 From: gathmann@scar.utoronto.ca (Oliver Gathmann) Date: Wed, 31 Mar 1999 12:10:35 -0500 (EST) Subject: [Matrix-SIG] ANN: PyDAS, the (Py)thon (D)ata (A)nalysis (S)ervant, version 0.1 Message-ID: This is to announce the avalability of the first public release of PyDAS, the Python Data Analysis Servant. Some time ago, this SIG indulged in the dream of, some day, having a full-fledged statistical analysis environment written in Python. Although the package I am announcing here is anything but full-fledged, I think it can at least serve as a prototype of some sort and stimulate thoughts on how the 'real thing' some day should look like. The package comes with a faunistical data set to play with (DB/Tables.shelf), a fact that has not (yet) found its way into the manual. What follows is the usual extract from the README file: -------------------------------------------------------------- What is PyDAS? PyDAS is a free program for interactive statistical analysis of data written in Python, a free, high-level scripting language. The main features of PyDAS are: * data browsing, editing, and plotting routines based on typed data tables (similar to data frames used in S-Plus) * specialized ordination and permutation procedures for statistical analysis of multivariate data; * a built-in Python console for interactive access of additional user-written data analysis routines; * a standardized set of high-level widgets for straightforward integration of user-written data analysis routines into the Graphical User Interface (GUI). Which platforms does it run on? All the software components used for PyDAS are known to run on Unix, Windows, and Mac systems. However, the current version has only been tested on Linux. Who would want it? * any experienced Python user looking for a convenient, GUI-assisted way of manipulating and plotting table-structured data; * any scientist with unusual statistical data analysis needs and some programming literacy looking for a tool that handles data input/output and that allows efficient integration of self-written data analysis routines; * any teacher in scientific computing looking for a free interactive programming environment that extends a powerful, yet easy to learn language (Python) with a convenient set of data manipulation routines. What is the current status of PyDAS? The current version of PyDAS (0.1) is the first public release and has to be considered as alpha code. Where is it located? The package and documentation are avalable at http://www.scar.utoronto.ca/~gathmann/PyDAS.html ---------------------------------------------------- Feedback very welcome, Oliver F. Oliver Gathmann (gathmann@scar.utoronto.ca) Surface and Groundwater Ecology Research Group University of Toronto phone: (416) - 287 7420 ; fax: (416) - 287 7423 web: http://www.scar.utoronto.ca/~gathmann