Mailman 3 June 2001 - NumPy-Discussion

simple array help?
by Pete Shinners 17 Sep '01

17 Sep '01

i have a simple array with floating values >>> print a array([ 0. , 0.2, 0.4, 0.6, 0.8]) what i need to do is change each index into another dimension with 3 indices. hmm, kind of confusing, an example will explain better >>> b = array(zip(a,a,a)) >>> print b array([[ 0. , 0. , 0. ], [ 0.2, 0.2, 0.2], [ 0.4, 0.4, 0.4], [ 0.6, 0.6, 0.6], [ 0.8, 0.8, 0.8]]) ok, this does the job, but once i start using large arrays, the back-n-forth conversion between arrays and python lists is costing me quite a bit. is there a way i can reshape the array this way without the call to python's 'zip'? (no, 'map' doesn't count either) i've tried much fooling with NewAxis, resize, and friends. but either i haven't stumbled upon the correct combination or i'm not trying the correct tools. i'd like to keep this as quick as possible, so hopefully it can be done without anything too elaborate. thanks (doh, sorry for the previous distutils related post... hit the wrong mailing list)

6 19

Numpy install problems
by Michael D. Risser 27 Jul '01

27 Jul '01

When attempting to install Numeric 17.0 with Python 1.5.2 and Distutils 1.0 on Red Hat 7.0, I get the following error after issuing the command 'python setup.py install' Traceback (innermost last): File "setup.py", line 15, in ? vs = map(string.atoi, v.groups()) AttributeError: 'None' object has no attribute 'groups' This error occurs regardless of whether or not I am root. I have also tried uninstalling the Red Hat Python packages and installing all from source, but I get the same error. Any help would be appreciated. Thanks.

4 5

MA : assignment to slices doesn't work like Numeric.
by Joe Van Andel 09 Jul '01

09 Jul '01

I retrieved the latest MA from CVS. I've noticed that assigning to a slice doesn't work the same as Numeric. Here's a simple test program: -------------------------------- from MA import * #from Numeric import * numBeams,numGates = (5,4) result = ones((numBeams, numGates),'f') * -327.68 print 'result = ', result t1 = ones((numGates,),'f') t2 = 2* ones((numGates,),'f') result[0] = t1 result[1][:] = t2 print 'result = ', result ----------------------------------------- Output using 'MA': result = [[-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,]] result = [[ 1. , 1. , 1. , 1. ,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,]] However, if I use Numeric, rather than MA, I get: result = [[-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68]] result = [[ 1. 1. 1. 1. ] [ 2. 2. 2. 2. ] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68]] So, a[x][:] = my_array doesn't seem to work using 'MA'. -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu

2 1

speed problem with Numeric sort
by hoel＠germanlloyd.org 19 Jun '01

19 Jun '01

We have a speed problem with Numeric.sort on large arrays with only a few different values. Here is my example -- snip -- >cat numtst.py import Numeric print Numeric.__version__ class timer: def __init__(self): import time self.start = time.time() def stop(self): import time print "%.3f" % (time.time() - self.start) from RandomArray import random from Numeric import sort, Int r=random((71400,)) t = timer() ; p=sort(r) ; t.stop() r=(random((71400,))*70000).astype(Int) t = timer() ; p=sort(r) ; t.stop() r=(random((71400,))*70).astype(Int) t = timer() ; p=sort(r) ; t.stop() r=(random((71400,))*7).astype(Int) t = timer() ; p=sort(r) ; t.stop() 16:27 hoel@seeve:hoel 2>python numtst.py 17.3.0 0.185 0.148 2.053 21.668 -- snip -- So the less different values are contained in the array the longer takes the sorting. Is this also the case with newer versions of Numeric (But this is Python 1.5.2)? Why is sorting of these arrays so slow? Thanks Berthold -- email: hoel(a)GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer.

3 4

MA improvements
by Paul F. Dubois 15 Jun '01

15 Jun '01

Heavy MA users may wish to get the latest from CVS to try. It has numerous improvements as detailed in changes.txt. I am sending out this announcement because I won't be available to work on it for three weeks, so if you find it is broken sync back to June 14 until I can fix it.

1 0

Experimental version of MA.py
by Paul F. Dubois 15 Jun '01

15 Jun '01

The version of MA.py checked in today supports more of the Numeric API. If you are a heavy MA user you may wish to check it out. Due to other commitments I will be unavailable for further work and testing on it for about 3 weeks. I attach the file for your convenience. To try it, drop it in the site-packages/MA directory in place of your current one. I also gave MA a new test routine very close to the new one for Numeric.

1 0

New testing procedures
by Paul F. Dubois 14 Jun '01

14 Jun '01

In CVS there is now a file Test/test.py instead of the previous test_items and test_all.py. test.py uses PyUnit, the new testing framework in Python. Developers should be able to add tests much more easily now.

1 0

performance comparison of C++ vs Numeric (MA) operations.
by Joe Van Andel 13 Jun '01

13 Jun '01

I was curious about the relative performance of C++ vs Numeric Python, for operations on arrays of roughly 400,000 array elements. I built a simple array single precision multiplication function in C++, that performs an element by element multiply, checking whether each element is "valid" or "missing data". Then, for comparision, I wrote a similar multiplication routine, using the Masked Array (MA) package of Numeric Python. I compiled Numeric Python (20.1.0b2) with '-O3', by modifying setup.py to contain lines like OPTIMIZE=['-O3'] ext_modules = . . Extension('multiarray', ['Src/multiarraymodule.c'], extra_compile_args=OPTIMIZE ), --------------------------------------- On an 800 Mhz dual processor Dell Linux box, using gcc 2.95.3, Software Performance ------------------------------------------------ Numeric Python 5.0e6 multiplies/second Numeric Python -03 6.1e6 multiplies/second C++ 10.3e6 multiplies/second C++ -O3 10.3e6 multiplies/second (I tried using "plain" Numeric arrays, rather than Masked arrays, and it didn't seem to make much difference.) Has anyone else benchmarked the relative performance of C/C++ vs Numeric Python? Does anyone know of other optimizations to Numeric Python that could be implemented? I know a more realistic benchmark would include I/O, which might tend to reduce the apparent difference in performance. I've attached the benchmark modules, in case someone would like to examine them. -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu import sys # test harness for Masked array performonce from MA import * #from Numeric import * from Perp.util.TimerUtility import TimerUtility def mult_test(a1, a2): res = a1 * a2 if __name__ == '__main__': repeat = 100 gates = 1000 beams = 370 if len(sys.argv) > 1: repeat = int(sys.argv[1]) t1 = ones((beams, gates), Float) a1 = masked_values(t1, -327.68) a2 = masked_values(t1, -327.68) i = 0 tu = TimerUtility(()) while (i < repeat): i = i+1 res = mult_test(a1, a2) elapsed = tu.elapsed() print 'completed %d in %f seconds' % (repeat , elapsed) cntMultiply = repeat*gates*beams print '%8.3g checked multiplies/second' % (cntMultiply/elapsed) #include <iostream> #include <stdlib.h> #include "PerfTimer.h" typedef float *FLOAT_PTR; extern void mult_test(FLOAT_PTR *a1, FLOAT_PTR *a2, FLOAT_PTR *resp, float missingValue); const int gates = 1000; const int beams = 370; int main(int argc, char *argv[]) { int repeat =100; const float missingValue = -327.68; if (argc > 1) repeat = atoi(argv[1]); FLOAT_PTR *a1 = new FLOAT_PTR[beams]; FLOAT_PTR *a2 = new FLOAT_PTR[beams]; FLOAT_PTR *res = new FLOAT_PTR[beams]; // allocate storage for 2d variables for (int b = 0; b < beams; ++b) { a1[b] = new float[gates]; a2[b] = new float[gates]; res[b] = new float[gates]; } PerfTimer pt; for (int r = 0; r < repeat; ++r) { mult_test(a1, a2, res,missingValue); } double elapsed = pt.Elapsed(); double cntMultiply = repeat*gates*beams; cout << repeat << " repetitions completed" << endl; cout << cntMultiply << "checked multiplies" << endl; cout << cntMultiply/elapsed << "checked multiplies/second" << endl; } void mult_test(FLOAT_PTR *a1, FLOAT_PTR *a2, FLOAT_PTR *resp, float missingValue) { const float atol = 1.e-8; const float rtol = 1.0e-5; for (int b=0; b < beams; ++b) { for (int g = 0; g < gates; ++g) { if (fabs(a1[b][g] - missingValue) < atol + rtol * fabs(missingValue)) { resp[b][g] = missingValue; continue; } else if (fabs(a2[b][g] - missingValue) < atol + rtol * fabs(missingValue)) { resp[b][g] = missingValue; } else { resp[b][g] = a1[b][g] * a2[b][g]; } } // for gates } // for beams } #include <time.h> class PerfTimer { public: // constructor, starts timing PerfTimer(); // reset starting time void Start(); // compute elapsed time since last construction or Start() double Elapsed(); private: struct timespec startTime_; }; #include "PerfTimer.h" PerfTimer::PerfTimer() { Start(); } void PerfTimer::Start() { clock_gettime(CLOCK_REALTIME, &startTime_); } double PerfTimer::Elapsed() { struct timespec stopTime; clock_gettime(CLOCK_REALTIME, &stopTime); return (stopTime.tv_sec + stopTime.tv_nsec/1.0e9 - (startTime_.tv_sec + startTime_.tv_nsec/1.0e9) ); } #CCFLAGS=-O3 CCFLAGS= CXXFLAGS=${CCFLAGS} all: arrayperf arrayperf: arrayperf.cc g++ -o arrayperf arrayperf.cc PerfTimer.cc -lm -lrt # import time class TimerUtility: """ Timer/Utility for performance measurment """ def __init__(self, verbose=1,useElapsed=0): """ ctor: can suppress printing by settings **verbose** to 0 """ # if we're timing elapsed events, including subprocesses, # then use time.time() if useElapsed: self.__timeFunc = time.time else: self.__timeFunc = time.clock self.__startTime = self.__timeFunc() self.__lastTime = self.__startTime self.__verbose = verbose def elapsed(self, msg = 'Elapsed '): """ print elapsed time since instance creation or last **elapsed()** call """ current = self.__timeFunc() delta = (current - self.__lastTime) if (self.__verbose): print '%s : %5.2f' % (msg, float(delta)) self.__lastTime = current return delta def rate(self, count, msg = 'rate'): """ print elapsed time and rate since instance creation or last **elapsed()** call """ current = self.__timeFunc() delta = (current - self.__lastTime) if (self.__verbose): print '%s : %5.2f : %6.2f' % (msg, float(delta), float(count)/float(delta)) self.__lastTime = current return delta def total(self, msg = 'Total '): """ print total time since TimerUtility was created """ current = self.__timeFunc() diff = (current - self.__startTime) if (self.__verbose): print '%s : %5.2f' % (msg, diff) return diff

3 3

PS: underflow array([2.9e-131+0j])**3 (why complex give a different result? bug?)
by Jean-Bernard Addor 12 Jun '01

12 Jun '01

Hey Numpy people! This a PS from my previous message. I just tried the underflow with complex : >>> Numeric.array([2.9e-131])**3 Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: math range error >>> 2.9e-131**3 0.0 >>> Numeric.array([2.9e-131+0j])**3 array([ 0.+0.j]) Now I have a quick solution for my problem, but I have the impression that this is a bug, at least I don't understand the underlying logic. If somebody can explain it to me? Jean-Bernard

2 1

underflow array([2.9e-131])**3
by Jean-Bernard Addor 12 Jun '01

12 Jun '01

Hey Numpy people! Do anyone know how to disable underflow exception errors in Numeric? I have a lot of these in my code. It is now a very important problem in my calculations. The only solution I see is to make a for loop and make the arithmetic in python instead of Numeric. Thanks for your help, Jean-Bernard Python 2.0 (#9, Feb 2 2001, 12:17:02) [GCC 2.95.2 19991024 (release)] on linux2 Type "copyright", "credits" or "license" for more information. Hello from .pythonrc.py >>> import Numeric >>> Numeric.__version__ '17.2.0' >>> Numeric.array([2.9e-131])**3 Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: math range error >>> 2.9e-131**3 0.0

2 1

Re: [Numpy-discussion] math range error with Masked Array
by Joe Van Andel 08 Jun '01

08 Jun '01

"Paul F. Dubois" wrote: > > Is the array single or double precision? Does it have the spacesaver > attribute set? The array is single precision, with the spacesave attribute set. > > Is it possible the data had bad values in it that were something other than > the missing value? What was the missing value? Yes, I finally found that the array had some contained some 'Nan' variables, because of an error in my earlier calculation. So, at this point, MA works for me. Thanks for your help. -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu

1 0

3d interpolation
by Robert.Denham＠dnr.qld.gov.au 07 Jun '01

07 Jun '01

I have a series of x,y,z irregular data points. I would like to create a 2-d array (surface) from this data, each cell in the array being an interpolated value based on the nearby z values. I was wondering if anyone had any experience or suggestions under python? The interpolation algorithm needn't be elaborate, kriging is definite overkill, I was thinking more along the lines of splines or even inverse distance weighting. Thanks Robert Denham Department of Natural Resources Queensland, Australia ************************************************************************ The information in this e-mail together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any form of review, disclosure, modification, distribution and/or publication of this e-mail message is prohibited. If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network. ************************************************************************

2 1

vtkStructuredPoints
by Karshi Hasanov 07 Jun '01

07 Jun '01

Hi, I wanna build *.vtk structured data file from an array A[i,j,k] which has a vector attributes. What's the right( or best) way of doing it using python? I do have the VTK User's Guide Book, but it didn't tell me much. Thanks

2 1

20.1.0b2
by Paul F. Dubois 06 Jun '01

06 Jun '01

20.1.0b2 is now available in tar.gz, .exe, .zip formats.

1 0

undefined symbols when importing lapack
by David H. Marimont 06 Jun '01

06 Jun '01

I just compiled and installed Numeric 20.1.0b1 using lapack and blas libraries. When I tried to import lapack_lite (after importing Numeric), I got this error: Traceback (most recent call last): File "<stdiimoprimporn>", line 1, in ? ImportError: /usr/lib/liblapack.so.3: undefined symbol: e_wsfe I'm using Python 2.1 on RH 7.1. I've had this problem before and have even seen postings to this list about related problems. But the solutions posted were over my head, so I've never been able to use the python interface to the lapack and blas libraries, which I really need. Does anyone have any advice, preferably pitched to someone who has limited compilation skills (i.e. at the the "configure, make, make install" level)? Thanks. David Marimont

2 3

math range error with Masked Array
by Joe Van Andel 06 Jun '01

06 Jun '01

I'm trying to use the MA package for numeric computations. Unfortunately, attempting to construct a masked array sometimes fails: masked_values(values, missingValue,savespace=1) File "/usr/lib/python2.1/site-packages/MA/MA.py", line 1299, in masked_values m = Numeric.less_equal(abs(d-value), atol+rtol*abs(value)) OverflowError: math range error The odd thing is that the floating point calculations that produced the input Numeric array didn't cause a math range error, but MA's attempt to find the 'missing' values does cause a range error. When I switched to Python2.1, I had to find and fix several overflow problems that didn't cause exceptions under Python1.5. For example, I had to use a "protected" exponentation routine to avoid overflow errors: MIN_EXP = -745 MAX_EXP = 709 def ProtExp(a): """ Protected Exponentiation calculation. Avoid Overflow error on large negative or positive arguments """ min_a = choose(less(a, MIN_EXP), (a, MIN_EXP)) return exp(choose(greater(min_a, MAX_EXP), (min_a, MAX_EXP)) ) ------------------- I'm concerned that the math exception handling for Python2.1 under x86 Linux makes it hard to get my work done. Any ideas on how to fix this error in MA? (I already tried masked_values(values, missingValue, rtol=1e-2,atol=1.e-4, savespace=1), which didn't help.) -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu

1 0

segfault when pickling PyObject arrays
by Tavis Rudd 05 Jun '01

05 Jun '01

Hi, I've been having difficultly pickling arrays with the type PyObject using Numeric. I haven't tried it with MA but I assume the same problem exists. This script works ===================================== from cPickle import dump, load from Numeric import array, PyObject def pickleIt(obj, fileName): fp = open(fileName, 'w') dump(obj, fp) fp.close def loadIt(fileName): fp = open(fileName, 'r') obj = load(fp) fp.close() return obj a = array(['abc', 'def', 'ghi'], PyObject) pickleIt(a, 'test.pickle') This script segfaults ==================================== # ... same imports and func defs as above b = loadIt() print b ==================================== I first noticed this when trying to pickle arrays constructed from lists of mx.DateTime objects. Numeric 19.1.0 Python 2.1 final Linux 2.2.18 Is this a reproduceable bug or something unique to my setup? Tavis

2 3

Re: [Numpy-discussion] Rich Comparisons and the Docs
by Chris Barker 01 Jun '01

01 Jun '01

"Paul F. Dubois" wrote: > The doc source is not available. I try to keep it up to date. If you would > tell me the section I overlooked I will fix it. That would be: Special Topics: Comparisons > As to logical and and or, they are not available for overload in the rich > comparisons schema. That's a Python decision. I figured as much. I just took a look at PEP 207: Rich Comparisons, and interestingly enough, the orignal version by David Asher addressed this issue (at least in terms of chaining comparisons, such as a > b >c). In the final PEP, I foound this: 5 In the current proposal, when A<B returns an array of elementwise comparisons, this outcome is considered non-Boolean, and its interpretation as Boolean by the shortcut operators raises an exception. David Ascher's proposal tries to deal with this; I don't think this is worth the additional complexity in the code generator. Instead of A<B<C, you can write (A<B)&(B<C). I'm curious about the statement "I don't think this is worth the additional complexity". It seems like being able to override the boolean operators would be VERY useful, and it seems odd how quickly this was glossed over. It is extremely common to combine a boolean operator with a a comparison. Is it time for a "rich boolean operators" PEP, or was this really rejected for good reason? By the way, it doesn't seem to raise an exception for me, but it does give a wrong (uninuitive anyway)answer: >>> a = array([1,2,3]) >>> b = array([3,2,1]) >>> c = array([3,3,3]) >>> a < b < c array([0, 1, 1]) Also, does (A<B)&(B<C) always work? Is this a reliable solution? What do the bitwise operators do with Numeric arrays? I guess the final question here is: Is it time for a "rich boolean operators" PEP, or was this really rejected for good reason? -- Christopher Barker, Ph.D. ChrisHBarker(a)home.net --- --- --- http://members.home.net/barkerlohmann ---@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Oil Spill Modeling ------ @ ------ @ ------ @ Water Resources Engineering ------- --------- -------- Coastal and Fluvial Hydrodynamics -------------------------------------- ------------------------------------------------------------------------

1 0