Mailman 3 November 2009 - NumPy-Discussion

boolean arrays
by Nils Wagner 27 Nov '09

27 Nov '09

Hi all, is the following behaviour correct >>> a = array(([True,True],[True,True])) >>> b = array(([False,False],[False,False])) >>> a+b array([[ True, True], [ True, True]]) I have expected False. Nils

6 7

using numpy.savetxt to save columns of numerical values and columns of text values
by Richared Beare 27 Nov '09

27 Nov '09

I have been unable to find a way of doing a very simple thing: saving data that contains both arrays of numerical values and arrays of string values, using savetxt in numpy. As a very simple example, suppose a is a numpy array of integers and b is one containing strings, e.g.: a = np.array([1,2,3]) b = np.array(['one', 'two', 'three']) The following call to savetxt: savetxt('test.txt', transpose((a,b)), fmt='%i %s') produces the following error: float argument required, not numpy.string_ I don't have any problems if I mix integers and floats in the format string, or save all values as strings, but I am completely unable to save formatted numerical values and strings in different columns using savetxt. There must be an easy solution! Thank you. Richard Beare

3 2

NumPy Histogram for Tentative NumPy Tutorial Questions
by Wayne Watson 26 Nov '09

26 Nov '09

I decided to try some example code from Subject. import numpy import pylab # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 mu, sigma = 2, 0.5 v = numpy.random.normal(mu,sigma,10000) # Plot a normalized histogram with 50 bins pylab.hist(v, bins=50, normed=1) # matplotlib version (plot) pylab.show() # Compute the histogram with numpy and then plot it (n, bins) = numpy.histogram(v, bins=50, normed=1) # NumPy version (no plot) pylab.plot(.5*(bins[1:]+bins[:-1]), n) pylab.show() After the histogram is displayed how do I get to the plot? Where is histogram described in some detail? Normalized? The histogram x-axis goes from 0 to 4.5. How does that happen? Is v is two dimensional? What if it's one dimensional? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: <www.speckledwithstars.net/>

3 8

Help making better use of numpy array functions
by mdekauwe 26 Nov '09

26 Nov '09

Hi I have written some code and I would appreciate any suggestions to make better use of the numpy arrays functions to make it a bit more efficient and less of a port from C. Any tricks are thoughts would be much appreciated. The code reads in a series of images, collects a running total if the value is valid and averages everything every 8 images. Code below... Thanks in advance... #!/usr/bin/env python """ Average the daily LST values to make an 8 day product... Martin De Kauwe 18th November 2009 """ import sys, os, glob import numpy as np import matplotlib.pyplot as plt def averageEightDays(files, lst, count, numrows, numcols): day_count = 0 # starting day - tag for output file doy = 122 # loop over all the images and average all the information # every 8 days... for fname in glob.glob(os.path.join(path, files)): current_file = fname.split('/')[8] year = current_file.split('_')[2][0:4] try: f = open(fname, 'rb') except IOError: print "Cannot open outfile for read", fname sys.exit(1) data = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) f.close() # if it is day 1 then we need to fill up the holding array... # copy the first file into the array... if day_count == 0: lst = data for i in xrange(numrows): for j in xrange(numcols): # increase the pixel count if we got vaild data (undefined = -999.0 if lst[i,j] > -900.: count[i,j] = 1 day_count += 1 # keep a running total of valid lst values in an 8 day sequence elif day_count > 0 and day_count <= 7: for i in xrange(numrows): for j in xrange(numcols): # lst valid pixel? if data[i,j] > -900.: # was the existing pixel valid? if lst[i,j] < -900.: lst[i,j] = data[i,j] count[i,j] = 1 else: lst[i,j] += data[i,j] count[i,j] += 1 day_count += 1 # need to average previous 8 days and write to a file... if day_count == 8: for i in xrange(numrows): for j in xrange(numcols): if count[i,j] > 0: lst[i,j] = lst[i,j] / count[i,j] else: lst[i,j] = -999.0 day_count = 0 doy += 8 lst, count = write_outputs(lst, count, year, doy) # it is possible we didn't have enough slices to do the last 8day avg... # but if we have more than one day we shall use it # need to average previous 8 days and write to a file... if day_count > 1: for i in xrange(numrows): for j in xrange(numcols): if count[i,j] > 0: lst[i,j] = lst[i,j] / count[i,j] else: lst[i,j] = -999.0 day_count = 0 doy += 8 lst, count = write_outputs(lst, count, year, doy) def write_outputs(lst, count, year, doy): path = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" try: of = open(os.path.join(path, outfile), 'wb') except IOError: print "Cannot open outfile for write", outfile sys.exit(1) outfile2 = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" try: of2 = open(os.path.join(path, outfile2), 'wb') except IOError: print "Cannot open outfile for write", outfile2 sys.exit(1) # empty stuff and zero 8day counts lst.tofile(of) count.tofile(of2) of.close() of2.close() lst = 0.0 count = 0.0 return lst, count if __name__ == "__main__": numrows = 332 numcols = 667 path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" lst = np.zeros((numrows, numcols), dtype=np.float32) count = np.zeros((numrows, numcols), dtype=np.int) averageEightDays('lst_scr_2006*.gra', lst, count, numrows, numcols) lst = 0.0 count = 0.0 averageEightDays('lst_scr_2007*.gra', lst, count, numrows, numcols) -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp265… Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 10

Type Inference
by Dan Yamins 25 Nov '09

25 Nov '09

Sometimes I need to convert object-type arrays to their "natural, real" type, without a priori knowing what that type is, e.g. the equivalent of: >>> Y = np.array(X.tolist()) where X is the object array. If X is naturally an array of ints, Y will be an int array, if X is naturally strings, then Y will be '|Sn' where n is the right string length, the right casting will be done if there are mixed types, etc... My question is: is there a faster way to do this, given that X is already an object array. One possible (though maybe not the only) way this might work is if there is a faster lower-level numpy type-inference function that doesn't actually do the conversion, but just reports the "right" type, the type that np.array() *would* convert to. Does such a thing exist? And if so, (let's call it 'np.typeinferer', hypothetically), would it then be faster to do something like: >>> Y = np.array(X,np.typeinferer(X)) by-passing the step of converting the list-ified version of X to the new type? Or is the bulk of the work being done by np.array() the type inference anyway, so that it just makes sense to use np.array() in the first place? Sorry if this question is unclear -- and thanks for all the help recently. best, Dan

2 1

The trunk has been branched for 1.4.0
by David Cournapeau 25 Nov '09

25 Nov '09

Hi, I have finally branched the trunk into the 1.4.x branch. I have disabled the C API for datetime, and fixed the C API/ABI numbers. At this point, you should avoid committing anything which is not a high priority bug in the branch. I will prepare a first rc1 based on the branch, cheers, David

1 0

Correlation function about a factor of 100 slower than matlab/mathcad ... but with fft even worse ?
by qubax＠gmx.at 25 Nov '09

25 Nov '09

The correlation of a large data (about 250k points) v can be checked via correlate(v,v,mode='full') and ought to give the same result as the matlab function xcorr(v) FFT might speed up the evaluation ... In my specific case: xcorr takes about 0.2 seconds. correlate takes about 70 seconds. fftconvolve takes about 400 seconds. on the irc channel a second person happened to run into the same problem using mathcad and data consisting of about 300k points: correl takes about 1 second correlate takes 127 seconds fftconvolve was aborded because it took to long These tests were checked and confirmed by two other persons on the irc channel. The computers involved were 32bit as well as 64bit machines. All 4 persons are sure that lapack/atlas libraries are properly installed. Could someone please investigate why correlate and especially fftconvolve are orders of magnitude slower? Should more details / sample data be required, please let me know. Thanks, q ---------------------- executed code: tic=time.time() cor_c1=correlate(c1data[:,1],c1data[:,1],mode='full') toc=time.time() tic=time.time() cor_c1=fftconvolve(c1data[:,1],c1data[:,1],mode='full') toc=time.time() xcorr(data) -- The king who needs to remind his people of his rank, is no king. A beggar's mistake harms no one but the beggar. A king's mistake, however, harms everyone but the king. Too often, the measure of power lies not in the number who obey your will, but in the number who suffer your stupidity.

2 2

f2py callback bug?
by James McEnerney 25 Nov '09

25 Nov '09

2 4

Python 2.6, NumPy on CentOS 5.3
by rkdelisle＠gmail.com 25 Nov '09

25 Nov '09

An application package that I have requires Python 2.6 and NumPy. I've installed Python 2.6 in a parallel manner as follows: NO modification of the core Python2.4 in /usr/bin has been done. Rather, I installed Python 2.6 under /opt/Python_2.6.4 and modified my user (not root) environment variables appropriately. The directory /opt/Python_2.6.4 was modified with chown to give me rwx access. To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to build. When I attempt to configure/build I receive various errors related to blas and lapack. The NumPy configuration is searching /usr/lib, /usr/lib64, /usr/local/lib, and /usr/local/lib64 for various blas, lapack, and atlas libraries. Within /usr/lib64 I do find a few lapack and blas and lapack libraries installed (libblas.so.3.1.1 and liblapack.so.3.1.1), but configure is not finding them. No atlas libraries are found, but my understanding is that these are deprecated anyway. As an alternative, I tried to install NumPy using the standard Python 2.4.3 using yum install NumPy but I receive an error saying that NumPy is obsoleted by PyNumeric. What?? PyNumeric is the precursor to NumPy. So even in the most basic instance, I cannot install NumPy because a deprecated library is seen as higher priority? Even given the generally out of date nature of CentOS this is unrealistic. Finally, I could try to build blas and lapack myself, but this seems to border on insanity. Any help is appreciated. -Kirk

5 6

Bug in rec.fromarrays ; plus one other possible bug
by Dan Yamins 25 Nov '09

25 Nov '09

Hi, I'm writing to report what looks like a two bugs in the handling of strings of length 0. (I'm using 1.4.0.dev7746, on Mac OSX 10.5.8. The problems below occur both for python 2.5 compiled 32-bit as well as python2.6 compiled 64-bit). Bug #1: A problem arises when you try to create a record array passing a type of '|S0'. >>> Cols = [['test']*10,['']*10] When not passing any dtype, this is created into a recarray with no problem: >>> np.rec.fromarrays(Cols) rec.array([('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', '')], dtype=[('f0', '|S4'), ('f1', '|S1')]) However, trouble arises when I try to pass a length-0 dtype explicitly. >>> d = np.dtype([('A', '|S4'), ('B', '|S')]) >>> np.rec.fromarrays(Cols,dtype=d) rec.array([('test', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', '')], dtype=[('A', '|S4'), ('B', '|S0')]) The same thing occurs if I cast to np arrays before passing to np.rec.fromarrays: >>> _arrays = [np.array(Cols[0],'|S4'),np.array(Cols[1],'|S')] [array(['test', 'test', 'test', 'test', 'test', 'test', 'test', 'test', 'test', 'test'], dtype='|S4'), array(['', '', '', '', '', '', '', '', '', ''], dtype='|S1')] >>> np.rec.fromarrays(_arrays,dtype=d) rec.array([('test', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', '')], dtype=[('A', '|S4'), ('B', '|S0')]) (Btw, why does np.array(['',''],'|S')) return an array with dtype '|S1'? Why not '|S0'? Are length-0 arrays being avoided explicitly? If so, why?) Bug #2: I'm not sure this is a bug, but it is annoying: np.dtype won't accept '|S0' as a type argument. >>> np.dtype('|S0') TypeError: data type not understood I have to do something like this: >>> d = np.dtype('|S') >>> d dtype('|S0') to get what I want. Is this intended? Regardless, this inconsistency also means that things like: >>> np.dtype(d.descr) can fail even when d is a properly constructed dtype object with a '|S0' type, which seems a little perverse. Am I just not supposed to be working with length-0 string columns, period? Thanks, Dan

3 3