Mailman 3 June 2001 - NumPy-Discussion

simple array help?
by Pete Shinners 17 Sep '01

17 Sep '01

i have a simple array with floating values >>> print a array([ 0. , 0.2, 0.4, 0.6, 0.8]) what i need to do is change each index into another dimension with 3 indices. hmm, kind of confusing, an example will explain better >>> b = array(zip(a,a,a)) >>> print b array([[ 0. , 0. , 0. ], [ 0.2, 0.2, 0.2], [ 0.4, 0.4, 0.4], [ 0.6, 0.6, 0.6], [ 0.8, 0.8, 0.8]]) ok, this does the job, but once i start using large arrays, the back-n-forth conversion between arrays and python lists is costing me quite a bit. is there a way i can reshape the array this way without the call to python's 'zip'? (no, 'map' doesn't count either) i've tried much fooling with NewAxis, resize, and friends. but either i haven't stumbled upon the correct combination or i'm not trying the correct tools. i'd like to keep this as quick as possible, so hopefully it can be done without anything too elaborate. thanks (doh, sorry for the previous distutils related post... hit the wrong mailing list)

6 19

Numpy install problems
by Michael D. Risser 27 Jul '01

27 Jul '01

When attempting to install Numeric 17.0 with Python 1.5.2 and Distutils 1.0 on Red Hat 7.0, I get the following error after issuing the command 'python setup.py install' Traceback (innermost last): File "setup.py", line 15, in ? vs = map(string.atoi, v.groups()) AttributeError: 'None' object has no attribute 'groups' This error occurs regardless of whether or not I am root. I have also tried uninstalling the Red Hat Python packages and installing all from source, but I get the same error. Any help would be appreciated. Thanks.

4 5

MA : assignment to slices doesn't work like Numeric.
by Joe Van Andel 09 Jul '01

09 Jul '01

I retrieved the latest MA from CVS. I've noticed that assigning to a slice doesn't work the same as Numeric. Here's a simple test program: -------------------------------- from MA import * #from Numeric import * numBeams,numGates = (5,4) result = ones((numBeams, numGates),'f') * -327.68 print 'result = ', result t1 = ones((numGates,),'f') t2 = 2* ones((numGates,),'f') result[0] = t1 result[1][:] = t2 print 'result = ', result ----------------------------------------- Output using 'MA': result = [[-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,]] result = [[ 1. , 1. , 1. , 1. ,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,] [-327.68,-327.68,-327.68,-327.68,]] However, if I use Numeric, rather than MA, I get: result = [[-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68]] result = [[ 1. 1. 1. 1. ] [ 2. 2. 2. 2. ] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68] [-327.68 -327.68 -327.68 -327.68]] So, a[x][:] = my_array doesn't seem to work using 'MA'. -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu

2 1

speed problem with Numeric sort
by hoel＠germanlloyd.org 19 Jun '01

19 Jun '01

We have a speed problem with Numeric.sort on large arrays with only a few different values. Here is my example -- snip -- >cat numtst.py import Numeric print Numeric.__version__ class timer: def __init__(self): import time self.start = time.time() def stop(self): import time print "%.3f" % (time.time() - self.start) from RandomArray import random from Numeric import sort, Int r=random((71400,)) t = timer() ; p=sort(r) ; t.stop() r=(random((71400,))*70000).astype(Int) t = timer() ; p=sort(r) ; t.stop() r=(random((71400,))*70).astype(Int) t = timer() ; p=sort(r) ; t.stop() r=(random((71400,))*7).astype(Int) t = timer() ; p=sort(r) ; t.stop() 16:27 hoel@seeve:hoel 2>python numtst.py 17.3.0 0.185 0.148 2.053 21.668 -- snip -- So the less different values are contained in the array the longer takes the sorting. Is this also the case with newer versions of Numeric (But this is Python 1.5.2)? Why is sorting of these arrays so slow? Thanks Berthold -- email: hoel(a)GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer.

3 4

MA improvements
by Paul F. Dubois 15 Jun '01

15 Jun '01

Heavy MA users may wish to get the latest from CVS to try. It has numerous improvements as detailed in changes.txt. I am sending out this announcement because I won't be available to work on it for three weeks, so if you find it is broken sync back to June 14 until I can fix it.

1 0

Experimental version of MA.py
by Paul F. Dubois 15 Jun '01

15 Jun '01

The version of MA.py checked in today supports more of the Numeric API. If you are a heavy MA user you may wish to check it out. Due to other commitments I will be unavailable for further work and testing on it for about 3 weeks. I attach the file for your convenience. To try it, drop it in the site-packages/MA directory in place of your current one. I also gave MA a new test routine very close to the new one for Numeric.

1 0

New testing procedures
by Paul F. Dubois 14 Jun '01

14 Jun '01

In CVS there is now a file Test/test.py instead of the previous test_items and test_all.py. test.py uses PyUnit, the new testing framework in Python. Developers should be able to add tests much more easily now.

1 0

performance comparison of C++ vs Numeric (MA) operations.
by Joe Van Andel 13 Jun '01

13 Jun '01

I was curious about the relative performance of C++ vs Numeric Python, for operations on arrays of roughly 400,000 array elements. I built a simple array single precision multiplication function in C++, that performs an element by element multiply, checking whether each element is "valid" or "missing data". Then, for comparision, I wrote a similar multiplication routine, using the Masked Array (MA) package of Numeric Python. I compiled Numeric Python (20.1.0b2) with '-O3', by modifying setup.py to contain lines like OPTIMIZE=['-O3'] ext_modules = . . Extension('multiarray', ['Src/multiarraymodule.c'], extra_compile_args=OPTIMIZE ), --------------------------------------- On an 800 Mhz dual processor Dell Linux box, using gcc 2.95.3, Software Performance ------------------------------------------------ Numeric Python 5.0e6 multiplies/second Numeric Python -03 6.1e6 multiplies/second C++ 10.3e6 multiplies/second C++ -O3 10.3e6 multiplies/second (I tried using "plain" Numeric arrays, rather than Masked arrays, and it didn't seem to make much difference.) Has anyone else benchmarked the relative performance of C/C++ vs Numeric Python? Does anyone know of other optimizations to Numeric Python that could be implemented? I know a more realistic benchmark would include I/O, which might tend to reduce the apparent difference in performance. I've attached the benchmark modules, in case someone would like to examine them. -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu import sys # test harness for Masked array performonce from MA import * #from Numeric import * from Perp.util.TimerUtility import TimerUtility def mult_test(a1, a2): res = a1 * a2 if __name__ == '__main__': repeat = 100 gates = 1000 beams = 370 if len(sys.argv) > 1: repeat = int(sys.argv[1]) t1 = ones((beams, gates), Float) a1 = masked_values(t1, -327.68) a2 = masked_values(t1, -327.68) i = 0 tu = TimerUtility(()) while (i < repeat): i = i+1 res = mult_test(a1, a2) elapsed = tu.elapsed() print 'completed %d in %f seconds' % (repeat , elapsed) cntMultiply = repeat*gates*beams print '%8.3g checked multiplies/second' % (cntMultiply/elapsed) #include <iostream> #include <stdlib.h> #include "PerfTimer.h" typedef float *FLOAT_PTR; extern void mult_test(FLOAT_PTR *a1, FLOAT_PTR *a2, FLOAT_PTR *resp, float missingValue); const int gates = 1000; const int beams = 370; int main(int argc, char *argv[]) { int repeat =100; const float missingValue = -327.68; if (argc > 1) repeat = atoi(argv[1]); FLOAT_PTR *a1 = new FLOAT_PTR[beams]; FLOAT_PTR *a2 = new FLOAT_PTR[beams]; FLOAT_PTR *res = new FLOAT_PTR[beams]; // allocate storage for 2d variables for (int b = 0; b < beams; ++b) { a1[b] = new float[gates]; a2[b] = new float[gates]; res[b] = new float[gates]; } PerfTimer pt; for (int r = 0; r < repeat; ++r) { mult_test(a1, a2, res,missingValue); } double elapsed = pt.Elapsed(); double cntMultiply = repeat*gates*beams; cout << repeat << " repetitions completed" << endl; cout << cntMultiply << "checked multiplies" << endl; cout << cntMultiply/elapsed << "checked multiplies/second" << endl; } void mult_test(FLOAT_PTR *a1, FLOAT_PTR *a2, FLOAT_PTR *resp, float missingValue) { const float atol = 1.e-8; const float rtol = 1.0e-5; for (int b=0; b < beams; ++b) { for (int g = 0; g < gates; ++g) { if (fabs(a1[b][g] - missingValue) < atol + rtol * fabs(missingValue)) { resp[b][g] = missingValue; continue; } else if (fabs(a2[b][g] - missingValue) < atol + rtol * fabs(missingValue)) { resp[b][g] = missingValue; } else { resp[b][g] = a1[b][g] * a2[b][g]; } } // for gates } // for beams } #include <time.h> class PerfTimer { public: // constructor, starts timing PerfTimer(); // reset starting time void Start(); // compute elapsed time since last construction or Start() double Elapsed(); private: struct timespec startTime_; }; #include "PerfTimer.h" PerfTimer::PerfTimer() { Start(); } void PerfTimer::Start() { clock_gettime(CLOCK_REALTIME, &startTime_); } double PerfTimer::Elapsed() { struct timespec stopTime; clock_gettime(CLOCK_REALTIME, &stopTime); return (stopTime.tv_sec + stopTime.tv_nsec/1.0e9 - (startTime_.tv_sec + startTime_.tv_nsec/1.0e9) ); } #CCFLAGS=-O3 CCFLAGS= CXXFLAGS=${CCFLAGS} all: arrayperf arrayperf: arrayperf.cc g++ -o arrayperf arrayperf.cc PerfTimer.cc -lm -lrt # import time class TimerUtility: """ Timer/Utility for performance measurment """ def __init__(self, verbose=1,useElapsed=0): """ ctor: can suppress printing by settings **verbose** to 0 """ # if we're timing elapsed events, including subprocesses, # then use time.time() if useElapsed: self.__timeFunc = time.time else: self.__timeFunc = time.clock self.__startTime = self.__timeFunc() self.__lastTime = self.__startTime self.__verbose = verbose def elapsed(self, msg = 'Elapsed '): """ print elapsed time since instance creation or last **elapsed()** call """ current = self.__timeFunc() delta = (current - self.__lastTime) if (self.__verbose): print '%s : %5.2f' % (msg, float(delta)) self.__lastTime = current return delta def rate(self, count, msg = 'rate'): """ print elapsed time and rate since instance creation or last **elapsed()** call """ current = self.__timeFunc() delta = (current - self.__lastTime) if (self.__verbose): print '%s : %5.2f : %6.2f' % (msg, float(delta), float(count)/float(delta)) self.__lastTime = current return delta def total(self, msg = 'Total '): """ print total time since TimerUtility was created """ current = self.__timeFunc() diff = (current - self.__startTime) if (self.__verbose): print '%s : %5.2f' % (msg, diff) return diff

3 3

PS: underflow array([2.9e-131+0j])**3 (why complex give a different result? bug?)
by Jean-Bernard Addor 12 Jun '01

12 Jun '01

Hey Numpy people! This a PS from my previous message. I just tried the underflow with complex : >>> Numeric.array([2.9e-131])**3 Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: math range error >>> 2.9e-131**3 0.0 >>> Numeric.array([2.9e-131+0j])**3 array([ 0.+0.j]) Now I have a quick solution for my problem, but I have the impression that this is a bug, at least I don't understand the underlying logic. If somebody can explain it to me? Jean-Bernard

2 1

underflow array([2.9e-131])**3
by Jean-Bernard Addor 12 Jun '01

12 Jun '01

Hey Numpy people! Do anyone know how to disable underflow exception errors in Numeric? I have a lot of these in my code. It is now a very important problem in my calculations. The only solution I see is to make a for loop and make the arithmetic in python instead of Numeric. Thanks for your help, Jean-Bernard Python 2.0 (#9, Feb 2 2001, 12:17:02) [GCC 2.95.2 19991024 (release)] on linux2 Type "copyright", "credits" or "license" for more information. Hello from .pythonrc.py >>> import Numeric >>> Numeric.__version__ '17.2.0' >>> Numeric.array([2.9e-131])**3 Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: math range error >>> 2.9e-131**3 0.0

2 1