Mailman 3 October 2000 - NumPy-Discussion

simple array help?
by Pete Shinners 17 Sep '01

17 Sep '01

i have a simple array with floating values >>> print a array([ 0. , 0.2, 0.4, 0.6, 0.8]) what i need to do is change each index into another dimension with 3 indices. hmm, kind of confusing, an example will explain better >>> b = array(zip(a,a,a)) >>> print b array([[ 0. , 0. , 0. ], [ 0.2, 0.2, 0.2], [ 0.4, 0.4, 0.4], [ 0.6, 0.6, 0.6], [ 0.8, 0.8, 0.8]]) ok, this does the job, but once i start using large arrays, the back-n-forth conversion between arrays and python lists is costing me quite a bit. is there a way i can reshape the array this way without the call to python's 'zip'? (no, 'map' doesn't count either) i've tried much fooling with NewAxis, resize, and friends. but either i haven't stumbled upon the correct combination or i'm not trying the correct tools. i'd like to keep this as quick as possible, so hopefully it can be done without anything too elaborate. thanks (doh, sorry for the previous distutils related post... hit the wrong mailing list)

6 19

Numpy install problems
by Michael D. Risser 27 Jul '01

27 Jul '01

When attempting to install Numeric 17.0 with Python 1.5.2 and Distutils 1.0 on Red Hat 7.0, I get the following error after issuing the command 'python setup.py install' Traceback (innermost last): File "setup.py", line 15, in ? vs = map(string.atoi, v.groups()) AttributeError: 'None' object has no attribute 'groups' This error occurs regardless of whether or not I am root. I have also tried uninstalling the Red Hat Python packages and installing all from source, but I get the same error. Any help would be appreciated. Thanks.

4 5

[Q]Best way for an array operation?
by Daehyok Shin 04 May '01

04 May '01

What is the best Numpy way for the following work? for i in range(len(x)): if x[i] > 0: y[i] = v[i] z[i] = u[i]+2 Daehyok Shin (Peter)

5 5

Re: [Numpy-discussion] Da Blas
by Frank Horowitz 02 Jan '01

02 Jan '01

At 3:14 PM -0700 14/9/00, <dubois(a)users.sourceforge.net> wrote: >Well, I certainly didn't mean to flame anyone in particular, so I'm sorry >that Frank was offended. Publicly apologized, and publicly accepted. I'm sorry I lost my cool. Ray Beausoleil and I are having a look at this too. He's coming at the problem from a Windows perspective, and I'm coming at it from a Linux/Unix perspective. It's going to be a little slow going for me at first, since it appears that I'm going to have to get my head around the distutils way of doing things.... Just in case it saves anyone else following a false lead, yes, there is a bug in the (Linux Mandrake 7.1; but probably any unix-ish box) compile line for building the lapack_lite.so shared library (i.e. what is generated on my box is "-L/usr/lib-llapack" which obviously is missing a blank before the "-llapack" substring). However, when I coerced the distutils system to get around that bug (by specifying "/usr/lib " with a trailing blank for the BLASLIBDIR and LAPACKLIBDIR variables in setup.py) the same problem (i.e. an "ImportError: /usr/lib/liblapack.so.3: undefined symbol: e_wsfe" in importing lapack_lite) ultimately manifested itself. So my advice is not to bother tracking that one down (although it probably should be reported as a bug in distutils, since adding that trailing blank algorithmically instead of in a user modifiable configuration string is the "right thing to do", TM.). I'm still puzzled by Thomas Breul's report of preloading an f2c library squashing the bug under RedHat 6.2(?). That does not work under Mandrake, although clearly there is some bloody symbol table missing from somewhere. The trouble is to find out where, and then to coerce distutils to deal with that in a portable way... Frank -- -- Frank Horowitz frank(a)ned.dem.csiro.au CSIRO-Exploration & Mining, PO Box 437, Nedlands, WA 6009, AUSTRALIA Direct: +61 8 9284 8431; FAX: +61 8 9389 1906; Reception: +61 8 9389 8421

4 4

quick optimization
by Pete Shinners 27 Nov '00

27 Nov '00

i've got a quick optimization for the arrayobject.c source. it speeds my usage of numpy up by about 100%. i've tested with other numpy apps and noticed a minimum of about 20% speed. anyways, in "do_sliced_copy", change out the following block: if (src_nd == 0 && dest_nd == 0) { for(j=0; j<copies; j++) { memcpy(dest, src, elsize); dest += elsize; } return 0; } with this slightly larger one: if (src_nd == 0 && dest_nd == 0) { switch(elsize) { case sizeof(char): memset(dest, *src, copies); break; case sizeof(short): for(j=copies; j; --j, dest += sizeof(short)) *(short*)dest = *(short*)src; break; case sizeof(long): for(j=copies; j; --j, dest += sizeof(int)) *(int*)dest = *(int*)src; break; case sizeof(double): for(j=copies; j; --j, dest += sizeof(double)) *(double*)dest = *(double*)src; break; default: for(j=copies; j; --j, dest += elsize) memcpy(dest, src, elsize); } return 0; } anyways, you can see it's no brilliant algorithm change, but for me, getting a free 2X speedup is a big help. i'm hoping something like this can get merged into the next releases? after walking through the numpy code, i was surprised how almost every function falls back to do_sliced_copy (guess that's why it's at the top of the source?). that made it a quick target for making optimization changes.

3 2

Re: Legal Notice
by Greg Ball 31 Oct '00

31 Oct '00

I think the way Gerard is interpreting this passage > Permission to use, copy, modify, and distribute this software and its > documentation for any purpose and without fee is hereby granted, is that you can DISTRIBUTE the software but you can't CHARGE A FEE. However this appears in the normal python license and many others and I am sure the correct interpretation is that you can USE, DISTRIBUTE, etc the software without PAYING A FEE (to the copyright holders.) It is free software, a XFree86 style license according to RMS, which is BSD-like but with no advertising clause. Gerard, the license is fine for distribution and as Joe said, similar to many other free software packages. -Greg Ball

1 0

Windows zip of Numerical Python 17.1.1 available
by Paul F. Dubois 31 Oct '00

31 Oct '00

Numerical 17.1.1 is available in both source and prebuilt Windows versions at: http://sourceforge.net/projects/numpy Windows users, simply extract into your top level Python 2.0 directory, typically c:\python20. Thank you to Pete Shinners for putting the Windows file together. It is too late to change for the 17 series but in future I will make releases that have whole numbers only as names and add files to that release as they become available. That makes it a little clearer that you should take the most recent file of the desired type.

1 0

Re: Legal Notice (Gerard Vermeulen (CRTBT))
by Joe Harrington 31 Oct '00

31 Oct '00

Gerard, Could you explain why you think that the phrase you quoted is restrictive? Lots of software comes with that phrase, and it certainly hasn't inhibited its wide promulgation. All it says is that you can't pass off this software as your own, which would be lying anyway. As far as I can tell (I'm not a lawyer, etc.), it's far less restrictive than, say, GPL. It seems very like the BSD license you suggest using instead. If you include the packages in a larger work, you can just say: This work includes the blah blah package, which bears the following license: "Permission to use, copy, modify, and distribute this software for any purpose without fee is hereby granted, provided that this entire notice is included in all copies of any software which is or includes a copy or modification of this software and in all copies of the supporting documentation for such software. " Note that the notice *doesn't* say you have to *copyright* the larger package under the terms of the notice or to the owner of the included package, it just says you have to include the notice, presumably so the recipient knows that the package is in there, who the owner of that component is, and that distributing that part is allowed. If you look at the manuals to most commercial Unices, you'll find several pages of such notices from the owners of the various packages included in the system. What's the problem? --jh--

1 0

Numeric-17.1.1 on IRIX 6.3
by Bernd Weyrauch 30 Oct '00

30 Oct '00

Hi, I tried to install Numeric-17.1.1 under IRIX 6.3 after installing Python 2.0 without optimization and the -g workaround. I got this error message: # /usr/local/bin/python setup.py install running install running build running build_py not copying Lib/ArrayPrinter.py (output up-to-date) not copying Lib/Numeric.py (output up-to-date) not copying Lib/Precision.py (output up-to-date) not copying Lib/UserArray.py (output up-to-date) not copying Lib/numeric_version.py (output up-to-date) running build_ext building '_numpy' extension skipping Src/_numpymodule.c (build/temp.irix-6.3-2.0/Src/_numpymodule.o up-to-date) skipping Src/arrayobject.c (build/temp.irix-6.3-2.0/Src/arrayobject.o up-to-date) skipping Src/ufuncobject.c (build/temp.irix-6.3-2.0/Src/ufuncobject.o up-to-date) ld -shared -all build/temp.irix-6.3-2.0/Src/_numpymodule.o build/temp.irix-6.3-2.0/Src/arrayobject.o build/temp.irix-6.3-2.0/Src/ufuncobject.o -o build/lib.irix-6.3-2.0/_numpy.so ld: ERROR 48: Cannot access registry file ./so_locations (No locks available) - ignored. ld: FATAL 51: Can't assign virtual addresses for _numpy.so within specified range. Please check your registry file ./so_locations. error: command 'ld' failed with exit status 4 Has anybody else encountered this problem? Thanks a lot! Bernd -- Bernd Weyrauch Marie-Curie-Str. 9 D-60439 Frankfurt am Main Institute of Biophysical Chemistry Tel. +49-(0)69-798-296-72 Frankfurt J. W. Goethe University Fax +49-(0)69-798-296-32 Biocenter, N230 e-mail bernd(a)bpc.uni-frankfurt.de http://www.biozentrum.uni-frankfurt.de/~bernd http://www.cthulhu.onlinehome.de

1 0

Legal Notice
by Gerard Vermeulen (CRTBT) 29 Oct '00

29 Oct '00

I like to propose a numpy-RPM to MandrakeSoft for future inclusion in their contrib-cdroms. Of course, MandrakeSoft requires that all license issues are cleared. I understand that the file Legal.htm applies to the packages MA and RNG. Is this true? I like to point out that the phrase "Permission to use, copy, modify, and distribute this software for any purpose without fee is hereby granted, provided that this entire notice is included in all copies of any software which is or includes a copy or modification of this software and in all copies of the supporting documentation for such software. " is very restrictive. It forbids the inclusion of MA and RNG on a commercial CD rom. Isn't it? It also forbids a consultant to use it for his business. But what about somebody getting paid for his work in a government research laboratory? Could you replace this by something more BSD-like? Best regards -- Gerard Vermeulen

1 0

Save use of savespace?!
by Janko Hauser 28 Oct '00

28 Oct '00

Hello, a question in the newsgroup got me thinking about the save use of the ``savespace'' flag in modules, which should be used by third party code. Think of this function as an example: def freezing(S,P): """Returns the freezing point temperature of seawater with a salinity S and at a pressure P. """ TF=(-.0575+1.710523e-3*_A.sqrt(_A.absolute(S))-2.154996e-4*S)*S-7.53e-4*P return TF This looks simple and one wants to use the savespace flag for S and P. So now the questions are rolling. What if S or P are scalars? Use asarray with the savespace flag. -> Error, this returns a copy of the array and not a reference, which is bad for array arguments. So use first asarray and then do asarray(S).savespace(1). Ok this works for the function, but we need to reset the flag at the end of the function. Also bad, what if S or P had the flag set before they entered the function? Ok I have now build a class, which actually keeps a record of these flags and has some helper methods to handle this record. The example function looks now: def freezing(S,P): SP=mkspacesaver(S,P) if not SP.isfloat(): raise ValueError, 'Arguments need to be of some float type' TF=(-.0575+1.710523e-3*_A.sqrt(_A.absolute(S))-2.154996e-4*S)*S-7.53e-4*P SP.free(); SP.clean(TF) return TF Note the test for float type, because if the user lazily entered an integer array for e.g. the pressure, the then set savespace flag would result in a very wrong answer. Although this seems to work for me, I do not know if this really presents a solution, as I actually need to put these lines into every similar function. The other way, to put the coeff. into an array has also problems, as I first need to find out of which type all arguments are and choose the highest one and it makes it more difficult to follow the coeff. in the formula. Any other suggestions, more general ways to handle argument testing? Is the savespace flag only suited for more closed environments, where one can assume more about the arguments? Class attached, __Janko class mkspacesaver: def __init__(self, *args): """Set for all arguments the savespace flag and make them to arrays. Keep track of this. At the end of the function, all flags can be reset, by a call to the method .free() """ self.refs = [] self.flags= [] for arg in args: try: arg.spacesaver() ref = arg except AttributeError: ref = afuncs.asarray(arg) self.flags.append(ref.spacesaver()) ref.savespace(1) self.refs.append(ref) def isfloat(self, index=None): """ Test if all refs have a float type, if a sequence is given for index, only the indexed refs are tested. """ if index: for i in index: if not self.refs[i].typecode() in ('f','d'): return 0 else: for i in range(len(self.refs)): if not self.refs[i].typecode() in ('f','d'): return 0 return 1 def free(self): for i in range(len(self.refs)): self.refs[i].savespace(self.flags[i]) def clean(self, *args): """Test if the arguments have the savespace flag set and set it to zero. Handle also scalars """ for arg in args: try: arg.savespace(0) except AttributeError: pass

1 0

cvs modules
by Johannes Zellner 24 Oct '00

24 Oct '00

Hello, there are three modules Numerical numpy numpy2 in the cvs repository. Could someone explain the differences please ? -- Johannes

1 0

COM/Numeric incompatibility
by John Lull 23 Oct '00

23 Oct '00

(Also posted to comp.lang.python) (Addendum -- Mark Hammond suggests this may be a threading issue -- OpenDatabase() apparently causes Python threads to be initialized. Does NumPy or lapack lite use threads for anything?) I'm working on a Windows application that needs to use both Microsoft DAO and Numeric, & have run into a problem between the two. The following code illustrates the problem: from win32com.client import Dispatch engine = Dispatch("DAO.DBEngine.35") db = engine.OpenDatabase('c:\\test.mdb') from Numeric import array m = array([[1., 2.], [3., 4.]]) y = array([2., 1.]) from LinearAlgebra import linear_least_squares print linear_least_squares(m, y) print 'success!' The problem is that the statement 'print linear_...' never completes. I step through in the debugger under pythonwin, & the lapack routine called by linear_least_squares() never returns. If I remove the statement 'db = engine...' it completes normally. If I make a call to linear_least_squares() *before* the database stuff, the later call to linear_least_squares() works properly. test.mdb is an empty database created by Access 97. Using another database seems to make no difference. I am using: Python 1.52 win32all-132 NumPy 16.1 All are standard distribution builds. OS is Win98SE. DAO.DBEngine.35 resolves to DAO350.dll, dated 4/27/98. That, in turn (I believe) is using msJet35.dll dated 4/23/99. Any ideas? Thanks. Regards, John

2 1

Re: comments on the Numerical Python manual
by L. Busby 19 Oct '00

19 Oct '00

Fred - I'm forwarding your mail on to numpy-discussion(a)lists.sourceforge.net, which I believe is the proper venue to deal with your comments. If the current Numerical Python manual directed you to support(a)icf.llnl.gov for support-related questions, that also needs to be updated. Thank you - Lee Busby ======================================================================= From: Fred Yankowski <fred(a)ontosys.com> To: support(a)icf.llnl.gov Subject: comments on the Numerical Python manual Greetings, I'm learning NumPy by working through the "Numerical Python" manual (Oct 9, 2000 version). So far the manual has been a great introduction to the software and I really appreciate the work of the authors in providing it. I have found a couple of problems in the text, as follows: On page 4 in section 4, it seems that >>> vector1 = array((1,2,4,5)) should be the following instead: >>> vector1 = array((1,2,3,4,5)) On page 48 in section 8, the translation table used in the "brightening" function acts instead to darken the image. I found that scaling proportional to a square-root function worked well: table = (arange(256)**.5 * (255**.5)).astype('b') One side note: I'm surprised that the coordinate system used by the NumTut.view function has X axis values increasing to the left. Is that standard for some graphing systems? I've never worked with such a system before. Or am I misunderstanding what I'm seeing with view? I'm basing this on inspection of view(arange(100)**2)) and view(arange(100)**.5). -- Fred Yankowski fred(a)OntoSys.com tel: +1.630.879.1312 Principal Consultant www.OntoSys.com fax: +1.630.879.1370 OntoSys, Inc 38W242 Deerpath Rd, Batavia, IL 60510, USA

1 0

RE: Possible bug (was Re: numpy, overflow, inf, ieee, and rich comparison)
by Huaiyu Zhu 17 Oct '00

17 Oct '00

On the issue of whether Python should ignore over/underflow on IEEE-enabled platforms: [Tim Peters] > That would stop the exception on exp() underflow, which is what you're > concerned about. It would also stop exceptions on exp() overflow, and on > underflow and overflow for all other math functions too. I doubt Guido will > ever let Python ignore overflow by default, #ifdef'ed or not. A semantic > change that jarring certainly won't happen for 2.0 (which is just a week > away). It can be argued that on IEEE enabled systems the proper thing to do for overflow is simply return Inf. Raising exception is WRONG. See below. [Guido van Rossum] > Incidentally, math.exp(800) returns inf in 1.5, and raises > OverflowError in 2.0. So at least it's consistent. That is not consistent at all. Suppose I'm plotting the curve f(x) where x include some singular points of f. In the first case the plot works with some portion of the curve clipped. In the second case it bombs. [Tim Peters] > Nothing like that will happen without a PEP first. I would like to see > *serious* 754 support myself, but that depends too on many platform experts > contributing real work (if everyone ran WinTel, I could do it myself > <wink>). [Guido van Rossum] > Bingo! > > 1.5.2 links with -lieee while 2.0 doesn't. Removing -lieee from the > 1.5.2 link line makes is raise OverflowError too. Adding it to the > 2.0 link line makes it return 0.0 for exp(-1000) and inf for > exp(1000). If using ieee is as simple as setting such a flag, there is no reason at all not to use it. Here are some more examples: Suppose you have done hours of computation on a problem. Just as you are about to get the result, you get an exception. Why? Because the residual error is too close to zero. Suppose you want to plot the curve of Gausian distribution. Oops, it fails. Because beyond a certain region the value is near zero. With these kinds of problems, vectorized numerical calculation becomes nearly impossible. How do you work in such an environment? You have to wrap every calculation in a try/except structure, and whenever there is an exception, you have to revert to elementwise operations. In practice this simply means Python would not be suitable for numerical work at all. What about the other way round? No problem. It is easy to write functions like isNaN, isInf, etc. With these one can raise exceptions in any place one want. It is even possible to raise exceptions if a matrix is singular to a certain precision, etc. The key point to observe here is that most numerical work involve more than one element. Some of them may be out of mahcine bounds but the whole thing could still be quite meaningful. Vice versa it is also quite possible that all elements are within bounds while the whole thing is meaningless. The language should never take over or subvert decisions based on numerical analysis. [Tim Peters] > Ignoring ERANGE entirely is not at all the same behavior as 1.5.2, and > current code certainly relies on detecting overflows in math functions. As Guido observed ERANGE is not generated with ieee, even for overflow. So it is the same behavior as 1.5.2. Besides, no correct numerical code should depend on exceptions like this unless the machine is incapable of handling Inf and NaN. Even in the cases where you do want to detect overflow, it is still wrong to use exceptions. Here's an example: x*log(x) approaches 0 as x approaches 0. If x==0 then log(x)==-Inf but 0*-Inf==NaN, not what one would want. But exception is the wrong tool to hangle this, because if x is an array, some of its element may be zero but other's may not. The right way to do it is something like def entropy(probability): p = max(probability, 1e-323) return p*log(p) [Tim Peters] > In no case can you expect to see overflow ignored in 2.0. You are proposing a dramatic change from the behavior of 1.5.2. This looks like to me to need a PEP and a big debate. It would break a LOT of numerical computations. [Thomas Wouters] > I remember the patch that did this, on SF. It was titled "don't link with > -lieee if it isn't necessary" or something. Not sure what it would break, > but mayhaps declaring -lieee necessary on glibc systems is the right fix ? > > (For the non-autoconf readers among us: the first snippet writes a test > program to see if the function '__fpu_control' exists when linking with > -lieee in addition to $LIBS, and if so, adds -lieee to $LIBS. The second > snippet writes a test program to see if the function '__fpu_control' > exists with the current collection of $LIBS. If it doesn't, it tries it > again with -lieee, > > Pesonally, I think the patch should just be reversed... The comment above > the check certainly could be read as 'Linux requires -lieee for correct > f.p. operations', and perhaps that's how it was meant. The patch as described seems to be based on flawed thinking. The numbers Inf and NaN are always necessary. The -lieee could only be unnecessary if the behavior is the same as IEEE. Obviously it isn't. So I second Thomas's suggestion. [Tim Peters] > If no progress is made on determining the true cause over the next few days, > I'll hack mathmodule.c to ignore ERANGE in the specific case the result > returned is a zero (which would "fix" your exp underflow problem without > stopping overflow detection). Since this will break code on any platform > where errno was set to ERANGE on underflow in 1.5.2, I'll need to have a > long discussion w/ Guido first. I *believe* that much is actually sellable > for 2.0, because it moves Python in a direction I know he likes regardless > of whether he ever becomes a 754 True Believer. [Guido van Rossum] > No, the configure patch is right. Tim will check in a change that > treats ERANGE with a return value of 0.0 as underflow (returning 0.0, > not raising OverflowError). What is the reason to do this? It looks like intetionally subverting ieee even when it is available. I thought Tim meant that only logistical problems prevent using ieee. If you do choose this route, please please please ignore ERANGE entirely, whether return value is zero or not. The only possible way that ERANGE could be useful at all is if it could be set independently for each element of an array, and if it behave as a warning instead of an exception, ie. the calculation would continue if it is not caught. Well, then, Inf and NaN serve this purpose perfectly. It is very reasonable to set errno in glibc for this; it is completely unreasonable to raise an exception in Python, because exceptions cannot be set to individual elements and they cannot be ignored. Huaiyu -- Huaiyu Zhu hzhu(a)users.sourceforge.net Matrix for Python Project http://MatPy.sourceforge.net

13 20

compile for application
by Pete Shinners 17 Oct '00

17 Oct '00

i have a python application which has a small extension module it needs. (written in C for speed). does anyone have an example setup.py script that will build the extension, but leave the finished module in the current directory (and not intall to PYTHONHOME?) thanks

1 0

ANNOUNCE: PyClimate 1.0
by Jon Saenz 16 Oct '00

16 Oct '00

Monday, 10/16/2000 Hello, all. We are making the first announce of the first stable release (1.0) of our package pyclimate, which presents some tools used for climate variability analysis and which make extensive use of Numerical Python and C. It is released under the GNU Public License. Changes from the previous release Version 1.0--October, 2000. 1) Improved tests. They are more accurate, reliable, informative, comprehensive and use less disk space. 2) The package compiles using distutils. This feature has been checked on FreeBSD, Linux and OSF platforms. 3) Some minor typos corrected in the documentation. 4) Added KPDF.c, a extension module to estimate univariate and multivariate kernel--based probability density functions. 5) Added a class to compute the vertical component of the curl of a vectorial field in diffoperators.py. 6) DCDFLIB.C is currently distributed with the package. 7) KZFilter.py has been converted into a general purpose LinearFilter.py which holds the basic operations of any linear filter. There are two different subclasses currently, the Lanczos filter and the previous Kolmogorov--Zurbenko filter, KZFilter.py. The user can define new filters just by redefining the filter coefficients subclassing LinearFilter. The package also contains (from release 0.0): IO functions ------------ -ASCII files (simple, but useful) -ncstruct.py: netCDF structure copier. From a COARDS compliant netCDF file, this module creates a COARDS compliant file, copying the needed attributes, comments, and so on in one call. Time handling routines ---------------------- * JDTime.py -> Some C/Python functions to convert from date to Scaliger's Julian Day and from Julian Day to date. We are not trying to replace mxDate, but addressing a different problem. In particular, this module contains a routine especially suited to handling monthly time steps for climatological use. * JDTimeHandler.py -> Python module which parses the units attribute of the time variable in a COARDS file and which offsets and scales adequately the time values to read/save date fields. Interface to DCDFLIB.C ---------------------- A C/Python interface to the free DCDFLIB.C library is provided. This library allows direct and inverse computations of parameters for several probability distribution functions like Chi^2, normal, binomial, F, noncentral F, and many many more. EOF analysis ------------ Empirical Orthogonal Function analysis based on the SVD decomposition of the data matrix and related functions to test the reliability/degeneracy of eigenvalues (truncation rules). Monte Carlo test of the stability of eigenvectors to temporal subsampling. SVD decomposition ----------------- SVD decomposition of the correlation matrix of two datasets, functions to compute the expansion coefficients, the squared cumulative covariance fraction and the homogeneous and heterogeneous correlation maps. Monte Carlo test of the stability of singular vectors to temporal subsampling. Multivariate digital filter --------------------------- Multivariate digital filter (high and low pass) based on the Kolmogorov-Zurbenko filter Differential operators on the sphere ------------------------------------ Some classes to compute differential operators (gradient and divergence) on a regular latitude/longitude grid. LICENSE ======= GNU General Public License Version 2. PREREQUISITES ============= To be able to use it, you will need: 1. Python ;-) 2. netCDF library 3.4 or later 3. Scientific Python, by Konrad Hinsen IF AND ONLY IF you really want to change the C code (JDTime.[hc] and pycdf.[hc]), then, you will also need SWIG. COMPILATION =========== Now, we use distutils. The installation is simpler. DOCUMENTATION ============= Postscript and PDF versions of the manual are included in the distribution. We are preparing an even better version of the documentation. AVAILABILITY ============ http://lcdx00.wm.lc.ehu.es/~jsaenz/pyclimate (Europe) http://pyclimate.zubi.net/ (USA) http://starship.python.net/crew/~jsaenz (USA) Any feedback from the users of the package will be really appreciated by the authors. Enjoy. Jon Saenz, jsaenz(a)wm.lc.ehu.es Juan Zubillaga, wmpzuesj(a)lg.ehu.es Jesus Fernandez, chus(a)wm.lc.ehu.es <P><A HREF="http://starship.python.net/crew/jsaenz/pyclimate">PyClimate 1.0</A> - Several routines for the analysis of climate variability. (16-Oct-00)

1 0

Re: Numpy-discussion digest, Vol 1 #115 - 4 msgs
by tbreuel＠parc.xerox.com 13 Oct '00

13 Oct '00

On Fri, 13 Oct 2000 numpy-discussion-request(a)lists.sourceforge.net wrote: > having Python you indeed have bad *and* incomplete 754 support on every 754 > platform it runs on. Well, it depends on what you mean by "bad *and* incomplete 754 support". IEEE recommends specific semantics for comparison operators in language bindings, but I think their recommendations are undesirable from a programming language and software engineering point of view. If you follow IEEE semantics, you prohibit optimizers from optimizing in many cases and it contradicts type system semantics in some languages. Worst of all, programs that rely on IEEE semantics (in particular, of comparison operators) give no lexical indication that they do so; they will simply do something wrong, like go into an infinite loop or terminate a loop prematurely. I think it's best to raise an exception for any expression involving a NaN, unless the programmer explicitly indicated that he wants a NaN result and IEEE semantics. Another good approach is to provide separate IEEE operators and leave the behavior of the built-in operators on IEEE special values explicitly undefined. This keeps people from relying on one behavior or the other. To me, the worst possible choice is to "implement IEEE semantics correctly", i.e., in the way the IEEE authors envisioned it. (Of course, IEEE is reasonably nice from a numerical point of view; I just think they overextended themselves when talking about language bindings). > (*) Quiz: *if* you can manage to trick Python into creating a NaN on your > particular 754 platform, does the Python expression NaN == 1.0 return true, > false, or raise an exception? Answer before you try it. Then try it on > enough 754 platforms until you give up trying to guess in advance. NaN == > NaN is predictable in Python, and is the one 754 feature Python guarantees > won't work correctly on any 754 platform (although I've heard that it loses > this predictability when run using NumPy's flavor of comparisons instead of > core Python's). This is just the sort of issue I was referring to. In many dynamic languages, the assumption is that pointer quality implies object equality. That's a perfectly reasonable semantic choice. While Python may not implement "754 correctly" in the sense of the (probably) Fortran-thinking IEEE floating point designers, I think Python's choice is correct and should not be altered. People who really want IEEE floating point comparison operations can use something like ieee.equal(a,b). That alerts the reader that something special is going on, it will fail on non-IEEE platforms (indicating that the code wouldn't work correctly there anyway), and it makes only the code that needs to pay the price for strict IEEE conformance. Cheers, Thomas.

1 0

Possible bug in LinearAlgebra module
by Aureli Soria Frisch 13 Oct '00

13 Oct '00

Hi, I have been working with the function (implmenting the Moore-Penrose generalized inverse) : generalized_inverse of the module LinearAlgebra. It seems to present a bug when operating on matrices of complex numbers. I want someone to confirm this point before submitting the source. Thanks in advance, Aureli ################################# Aureli Soria Frisch Fraunhofer IPK Dept. Pattern Recognition post: Pascalstr. 8-9, 10587 Berlin, Germany e-mail:aureli@ipk.fhg.de fon: +49 30 39 00 61 50 fax: +49 30 39 17 517 #################################

4 3

RE: [Numpy-discussion] RE: Possible bug (was Re: numpy, overflow, inf, ieee, and rich comparison)
by Harrison, Robert J 12 Oct '00

12 Oct '00

I watch but rarely contribute to these dicussions, but I feel compelled to whole heartedly support Tim Peters comments regarding full 754 support with consistent cross platform behaviour. Like Tim I've being doing numerical computing for over two decades and IEEE is an indispensible standard. Yes, we usually disable most exception handling within performance critical kernels, but within robust solvers or modern linear algebra packages, full IEEE exception handling is vital. As Tim has said, full 754 will satisfy all users to the maximum extent possible. Robert -----Original Message----- From: Tim Peters [mailto:tim_one@email.msn.com] Sent: Wednesday, October 11, 2000 7:44 PM To: Huaiyu Zhu Cc: python-list(a)python.org; PythonDev; Numpy-Discussion; dubois(a)users.sourceforge.net Subject: [Numpy-discussion] RE: Possible bug (was Re: numpy, overflow, inf, ieee, and rich comparison) [Huaiyu Zhu] > On the issue of whether Python should ignore over/underflow on > IEEE-enabled platforms: > > It can be argued that on IEEE enabled systems the proper thing to do for > overflow is simply return Inf. Raising exception is WRONG. See below. Python was not designed with IEEE-754 in mind, and *anything* that happens wrt Infs, NaNs, and all other 754 features in Python is purely an accident that can and does vary wildly across platforms. And, as you've discovered in this case, can vary wildly also across even a single library, depending on config options. We cannot consider this to be a bug since Python has had no *intended* behavior whatsoever wrt 754 gimmicks. We can and do consider gripes about these accidents to be feature requests. [Guido] > Incidentally, math.exp(800) returns inf in 1.5, and raises > OverflowError in 2.0. So at least it's consistent. [Huaiyu] > That is not consistent at all. Please read with an assumption of good faith. Guido was pointing out that-- all in the specific case of gcc+glibc on Linux (these don't hold on other platforms) --exp(800) returning Inf in 1.5 and OverflowError in 2.0 is consistent *with* that exp(-800) returns 0 in 1.5 and raises an exception in 2.0. He's right; indeed, he is in part agreeing with you. [Guido > 1.5.2 links with -lieee while 2.0 doesn't. Removing -lieee from the > 1.5.2 link line makes is raise OverflowError too. Adding it to the > 2.0 link line makes it return 0.0 for exp(-1000) and inf for > exp(1000). [Huaiyu] > If using ieee is as simple as setting such a flag, there is no > reason at all not to use it. The patch to stop setting -lieee was contributed by a Python user who claimed it fixed bugs on *their* platform. That's "the reason". We don't want to screw them either. > Here are some more examples: > ... I understand that 754 semantics can be invaluable. So does Guido. There's no argument about that. But Python doesn't yet support them, and wishing it did doesn't make it so. If linking with -lieee happens to give you the semantics you want on your platform today, you're welcome to build with that switch. It appears to be a bad choice for *most* Python users, though (more below), so we don't want to do it in the std 2.0 distro. > ... > In practice this simply means Python would not be suitable for numerical > work at all. Your biggest obstacle in getting Python support for 754 will in fact be opposition from number-crunchers. I've been slinging fp for 21 years, 15 of those writing compilers and math libraries for "supercomputer" vendors. 754 is a Very Good Thing, but is Evil in subset form (see Kahan's (justified!) vilification of Java on this point); ironically, 754 is hardest to sell to those who could benefit from it the most. > What about the other way round? No problem. It is easy to write > functions like isNaN, isInf, etc. It's easy for a platform expert to write such functions for their specific platform of expertise, but it's impossible to write them in a portable way before C99 is implemented (C99 requires that library suppliers provide them, rendering the question moot). > ... > The language should never take over or subvert decisions based on > numerical analysis. Which is why a subset of 754 is evil. 754 requires that the user be able to *choose* whether or not, e.g., overflow signals an exception. Your crusade to insist that it never raise an exception is as least as bad (I think it's much worse) as Python's most common accidental behavior (where overflow from libm usually raises an exception). One size does not fit all. [Tim] > Ignoring ERANGE entirely is not at all the same behavior as 1.5.2, and > current code certainly relies on detecting overflows in math functions. [Huaiyu] > As Guido observed ERANGE is not generated with ieee, even for > overflow. So it is the same behavior as 1.5.2. You've got a bit of a case of tunnel vision here, Huaiyu. Yes, in the specific case of gcc+glibc+Linux, ignoring ERANGE returned from exp() is what 1.5.2 acted like. But you have not proposed to ignore it merely from ERANGE returned from exp() in the specific case of gcc+glibc+Linux. This code runs on many dozens of platforms, and, e.g., as I suggested before, it looks like HPUX has always set errno on both overflow and underflow. MS Windows sets it on overflow but not underflow. Etc. We have to worry about the effects on all platforms. > Besides, no correct numerical code should depend on exceptions like > this unless the machine is incapable of handling Inf and NaN. Nonsense. 754 provides the option to raise an exception on overflow, or not, at the user's discretion, precisely because exceptions are sometimes more appropriate than Infs of NaNs. Kahan himself isn't happy with Infs and NaNs because they're too often too gross a clue (see his writings on "presubstitution" for a more useful approach). [Tim] > In no case can you expect to see overflow ignored in 2.0. [Huaiyu] > You are proposing a dramatic change from the behavior of 1.5.2. > This looks like to me to need a PEP and a big debate. It would break > a LOT of numerical computations. I personally doubt that, but realize it may be true. That's why he have beta releases. So far yours is the only feedback we've heard (thanks!), and as a result we're going to change 2.0 to stop griping about underflow, and do so in a way that will actually work across all platforms. We're probably going to break some HPUX programs as a result; but they were relying on accidents too. [Guido] > No, the configure patch is right. Tim will check in a change that > treats ERANGE with a return value of 0.0 as underflow (returning 0.0, > not raising OverflowError). [Huaiyu] > What is the reason to do this? It looks like intetionally subverting > ieee even when it is available. I thought Tim meant that only logistical > problems prevent using ieee. Python does not support 754 today. Period. I would like it to, but not in any half-assed, still platform-dependent, still random crap-shoot, still random subset of 754 features, still rigidly inflexible, way. When it does support it, Guido & I will argue that it should enable (what 754 calls) the overflow, divide-by-0 and invalid operation exceptions by default, and disable the underflow and inexact exceptions by default. This ensures that, under the default, an infinity or NaN can never be created from non-exceptional inputs without triggering an exception. Not only is that best for newbies, you'll find it's the *only* scheme for a default that can be sold to working number-crunchers (been there, done that, at Kendall Square Research). It also matches Guido's original, pre-754, *intent* for how Python numerics should work by default (it is, e.g., no accident that Python has always had an OverflowError exception but never an UnderflowError one). And that corresponds to the change Guido says <wink> I'm going to make in mathmodule.c: suppress complaints about underflow, but let complaints about overflow go through. This is not a solution, it's a step on a path towards a solution. The next step (which will not happen for 2.0!) is to provide an explicit way to, from Python, disable overflow checking, and that's simply part of providing the control and inquiry functions mandated by 754. Then code that would rather deal with Infs than exceptions can, at its explicit discretion. > If you do choose this route, please please please ignore ERANGE entirely, > whether return value is zero or not. It should be clear that isn't going to happen. I realize that triggering overflow is going to piss you off, but you have to realize that not triggering overflow is going to piss more people off, and *especially* your fellow number-crunchers. Short of serious 754 support, picking "a winner" is the best we can do for now. You have the -lieee flag on your platform du jour if you can't bear it. [Paul Dubois] > I don't have time to follow in detail this thread about changed behavior > between versions. What makes you think we do <0.5 wink>? > These observations are based on working with hundreds of code authors. I > offer them as is. FWIW, they exactly match my observations from 15 years on the HW vendor side. > a. Nobody runs a serious numeric calculation without setting underflow-to- > zero, in the hardware. You can't even afford the cost of software checks. > Unfortunately there is no portable way to do that that I know of. C allows libm implementations a lot of discretion in whether to set errno to ERANGE in case of underflow. The change we're going to make is to ignore ERANGE in case of underflow, ensuring that math.* functions will *stop* raising underflow exceptions on all platforms (they'll return a zero instead; whether +0 or -0 will remain a platform-dependent crap shoot for now). Nothing here addresses underflow exceptions that may be raised by fp hardware, though; this has solely to do with the platform libm's treatment of errno. So in this respect, we're removing Python's current unpredictability, and in the way you favor. > b. Some people use Inf but most people want the code to STOP so they can > find out where the INFS started. Otherwise, two hours later you have big > arrays of Infs and no idea how it happened. Apparently Python on libc+glibc+Linux never raised OverflowError from math.* functions in 1.5.2 (although it did on most other platforms I know about). A patch checked in to fix some other complaint on some other Unixish platform had the side-effect of making Python on libc+glibc+Linux start raising OverflowError from math.* functions in 2.0, but in both overflow and underflow cases. We intend to (as above) suppress the underflow exceptions, but let the overflow cases continue to raise OverflowError. Huaiyu's original complaint was about the underflow cases, but (as such things often do) expanded beyond that when it became clear he would get what he asked for <wink>. Again we're removing Python's current unpredictability, and again in the way you favor -- although this one is still at the mercy of whether the platform libm sets errno correctly on overflow (but it seems that most do these days). > Likewise sqrt(-1.) needs to stop, not put a zero and keep going. Nobody has proposed changing anything about libm domain (as opposed to range) errors (although Huaiyu probably should if he's serious about his flavor of 754 subsetism -- I have no idea what gcc+glibc+Linux did here on 1.5.2, but it *should* have silently returned a NaN (not a zero) without setting errno if it was *self*-consistent -- anyone care to check that under -lieee?: import math math.sqrt(-1) NaN or ValueError? 2.0 should raise ValueError regardless of what 1.5.2 did here.). just-another-day-of-universal-python-harmony-ly y'rs - tim _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion

1 0

Re: Numpy-discussion digest, Vol 1 #112 - 8 msgs
by Greg Kochanski 12 Oct '00

12 Oct '00

Python has *got* to ignore underflow exceptions. Otherwise, virtually any numeric calculation that subtracts numbers will fail for no good reason. Worse, it will fail in a very sporadic, data-dependent manner. With double precision numbers, the failures will be rare, but anyone doing computations with floats will see unwanted exceptions with noticeable frequency. I once did a numeric calculation (a SVD classifier) on a system with IEEE-754 underflow exceptions turned on, and I lost a good square inch of hair and a solid week of life because of it. At one point, the numbers were converted from doubles to floats, and about 1 in every 10,000,000 of the numbers were too small to represent as a float. So, in about 1 in ten runs (an hour each), the code would crash for no obvious reason. It was a debugging nightmare. If python catches underflows, I'm going back to FORTRAN. On less crucial topics, I'm strongly in favor of preserving NaN and Inf. If I want my code to crash when some computation goes awry, I'll use assert, and crash it myself. Any serious numeric code should be loaded with assertions (any code at all, in fact). Asking the interpreter to do it for you is a crude and blunt instrument. If I may ask, what kind of platforms are there where people do math where the hardware *won't* support IEEE-754?

3 2

Re: [Numpy-discussion] Re: numpy, overflow, inf, ieee, and rich comparison
by L. Busby 11 Oct '00

11 Oct '00

It was an optional module (fpectl) that got added into Python 1.5. Looks like it has been carried forward pretty much unchanged into the 2.0 release candidate. To use the facility, you need to build python with --with-fpectl, then identify "dangerous" (likely to generate SIGFPE) operations, and surround them with a pair of macros PyFPE_START_PROTECT and PyFPE_END_PROTECT. This has the effect of turning any SIGFPE into a Python exception. Start with Include/pyfpe.h, look for example usage in Objects/floatobject.c. Grep the python source for FPE_ to find the several places where these hooks are located. [ Paul Dubois wrote ] >About controlling floating-point behavior with Numpy: I think that somewhere >buried in the sources Lee Busby had us all set if we would just go through >the source and stick in some macro in the right places (this was maybe 3 >years ago, hence the accurate and detailed memory dump) but it was on the >todo list so long it outlived the death of the todo list. > >Anyway, my recollection is if we ever did it then we would have control when >things like overflow happen. But we haven't. My last experience of this sort >of thing was that it was completely a hardware-dependent thing, and you had >to find out from each manufacturer what their routine was and how to call it >or what compiler flag to use. > >Well, sorry to be so imprecise but I thought it worth mentioning that some >steps had been taken even if I don't remember what they were.

1 0

Re: numpy, overflow, inf, ieee, and rich comparison
by Paul F. Dubois 11 Oct '00

11 Oct '00

About controlling floating-point behavior with Numpy: I think that somewhere buried in the sources Lee Busby had us all set if we would just go through the source and stick in some macro in the right places (this was maybe 3 years ago, hence the accurate and detailed memory dump) but it was on the todo list so long it outlived the death of the todo list. Anyway, my recollection is if we ever did it then we would have control when things like overflow happen. But we haven't. My last experience of this sort of thing was that it was completely a hardware-dependent thing, and you had to find out from each manufacturer what their routine was and how to call it or what compiler flag to use. Well, sorry to be so imprecise but I thought it worth mentioning that some steps had been taken even if I don't remember what they were.

1 0

version.py vs numeric_version.py
by Fred Yankowski 10 Oct '00

10 Oct '00

I couldn't seem to download a prebuilt distribution of NumPy from SourceForge earlier today, so I connected via CVS and grabbed the latest code. I was eventually able to build and install it OK, but I had to copy Lib/version.py to Lib/numeric_version.py since the latter didn't exist but other code refers to it. Just a "heads up"... -- Fred Yankowski fred(a)OntoSys.com tel: +1.630.879.1312 Principal Consultant www.OntoSys.com fax: +1.630.879.1370 OntoSys, Inc 38W242 Deerpath Rd, Batavia, IL 60510, USA

1 0

ANNOUNCE: PySymbolic - Doing Symbolics in Python
by Pearu Peterson 10 Oct '00

10 Oct '00

Hi! I have started a project PySymbolic - "Doing Symbolics in Python" PySymbolic is a collection of tools for doing elementary symbolic calculations in Python. The project is in alpha stage. Current features include basic simplifications: *** removing redundant parenthesis: (a) -> a *** applying factorization: -(a) -> -a, +a -> a, ~(a) -> ~a *** collecting terms: n*a+b+m*a -> k*a+b (where k=n+m) *** collecting powers: a**c*b/a**d -> a**(c-d)*b *** cancellations: 0*a -> 0, a+0 -> 0, a**0 -> 1, 1*a -> a, a**1 -> a *** rational arithmetic: 3/4+2/6-1 -> 1/12 *** opening parenthesis: (a+b)*c -> a*c+b*c *** expanding integer powers: (a+b)**n -> a**n+a**(n-1)*b+... *** sorting terms in alphabethic order: a+c+b -> a+b+c *** (more to come) Projects homepage is http://cens.ioc.ee/projects/pysymbolic/ You can download the source from http://cens.ioc.ee/projects/pysymbolic/PySymbolic.tgz Regards, Pearu

1 0

[ANNOUNCE] Ninth International Python Conference
by Paul F. Dubois 09 Oct '00

09 Oct '00

CALL FOR PAPERS, POSTERS, AND PARTICIPATION Ninth International Python Conference March 5-8, 2001 Long Beach, California Web site: http://python9.org The 9th International Python Conference (Python 9) is a forum for Python users, software developers, and researchers to present current work, discuss future plans for the language and commercial applications, and learn about interesting uses of Python. The conference will consist of a day of tutorials, two days of refereed paper and application tracks including a Zope track and a multi-technology Python Applications Track, a developers' day, a small exhibition, demonstrations, and posters. Paper submission deadline: Monday, November 6, 2000 Notification of acceptance: Monday, December 11, 2000 Final papers due: Monday, January 15, 2001 Authors are invited to submit papers for the Refereed Paper Track that: * Present new and useful applications and tools that utilize Python * Describe the use of Python in large, mission-critical or unusual applications * Address practical programming problems, and provide lessons based on experience for Python programmers Also sought are poster presentations of interesting work in progress. Full information is available on the website, http://python9.org

1 0