Mailman 3 April 2007 - NumPy-Discussion

Help using numPy to create a very large multi dimensional array
by Bruno Santos 18 Apr '07

18 Apr '07

Dear Sirs, I'm trying to use Numpy to solve a speed problem with Python, I need to perform agglomerative clustering as a first step to k-means clustering. My problem is that I'm using a very large list in Pyhton and the script is taking more than 9minutes to process all the information, so I'm trying to use Numpy to create a matrix. I'm reading the vectors from a text file and I end up with an array of 115*2634 float elements, How can I create this structure with numpy? Where is my code in python: #Read each document vector to a matrix doclist = [] matrix = [] list = [] for line in vecfile: list = line.split() for elem in range(1, len(list)): list[elem] = float(list[elem]) matrix.append (list[1:]) vecfile.close() #Read the desired number of final clusters numclust = input('Input the desired number of clusters: ') #Clustering process clust = rows ind = [-1, -1] list_j=[] list_k=[] while (clust > numclust): min = 2147483647 print('Number of Clusters %d \n' % clust) #Find the 2 most similares vectors in the file for j in range(0, clust): list_j=matrix[j] for k in range(j+1, clust): list_k=matrix[k] dist=0 for e in range(0, columns): result = list_j[e] - list_k[e] dist += result * result if (dist < min): ind[0] = j ind[1] = k min = dist #Combine the two most similaires vectores by median for e in range(0, columns): matrix[ind[0]][e] = (matrix[ind[0]][e] + matrix[ind[1]][e]) / 2.0 clust = clust -1 #Move up all the remaining vectors for k in range(ind[1], (rows - 1)): for e in range(0, columns): matrix[k][e]=matrix[k+1][e]

5 6

NumPy benchmark
by rex 18 Apr '07

18 Apr '07

I'm about to build numpy using Intel's MKL 9.1 beta and want to compare it with the version I built using MKL 8.1. Is the LINPACK benchmark the most appropriate? Thanks, -rex -- Pollytheism: n., the belief that there are many gods, all of them parrots.

2 5

Matlab Translation - sqrt elementwise
by Miquel Poch 17 Apr '07

17 Apr '07

Hi, I've found the next expression write it in Matlab, Rtx = sqrt(Rt); Rtx is a matrix, and that's why I need sqrt() to operate elementwise. I've read NumPy tutorial, and I know it's possible, A set of this functions, has been provided wich optimize certain kinds of calculations on arrays. Most of this functions, such as sin, cos and sqrt are unary functions wich operate elementwise. [Numerical Python, pg. 13] but I don't know who to do it. The next error appear when I execute the code, ''' exceptions.TypeError : only length-1 arrays can be converted to Python scalars '' Thanks in advance, Mike <numpy-discussion(a)scipy.org>

4 3

Fastest distance matrix calc
by Bill Baxter 17 Apr '07

17 Apr '07

I think someone posted some timings about this before but I don't recall. The task is to compute the matrix D from two sets of vectors x (M,d) and y (N,d). The output should be D where D[i,j] is norm(x[i]-y[j]) The Matlab NetLab toolkit uses something like this to compute it: d2 = (x*x).sum(1)[:,numpy.newaxis] + (y*y).sum(1) - 2*mult(x,y.T) And then does d2[d2<0]=0 because roundoff in the above can sometimes create negative values. And finally d = sqrt(d2) But it just doesn't seem like the best way to do it. Whatever was posted before I don't remember anything about a subtract.outer solution. Seems like something along the lines of (subtract.outer(x,y)**2).sum(axis=0) might be faster, and also avoid the need to check for negatives. I'll do some timings if no one's already done it. Just wanted to check first. --bb

8 10

Re: [Numpy-discussion] NumPy benchmark
by Ray Schumacher 17 Apr '07

17 Apr '07

I'm still curious about the licensing aspects of using Intel's compiler and libs. Is the compiled Python/numpy result distributable, like any other compiled program? Ray

2 1

ANN: SciPy 2007 Conference
by Travis Vaught 17 Apr '07

17 Apr '07

Greetings, The *SciPy 2007 Conference* has been scheduled for mid-August at CalTech. http://www.scipy.org/SciPy2007 Here's the rough schedule: Tutorials: August 14-15 (Tuesday and Wednesday) Conference: August 16-17 (Thursday and Friday) Sprints: August 18 (Saturday) Exciting things are happening in the Python community, and the SciPy 2007 Conference is an excellent opportunity to exchange ideas, learn techniques, contribute code and affect the direction of scientific computing (or just to learn what all the fuss is about). Last year's conference saw a near-doubling of attendance to 138, and we're looking forward to continued gains in participation. We'll be announcing the Keynote Speaker and providing a detailed schedule in the coming weeks. Registration: ------------- Registration is now open. You may register online at https://www.enthought.com/scipy07. Early registration for the conference is $150.00 and includes breakfast and lunch Thursday & Friday and a very nice dinner Thursday night. Tutorial registration is an additional $75.00. After July 15, 2007, conference registration will increase to $200.00 (tutorial registration will remain the same at $75.00). Call for Presenters ------------------- If you are interested in presenting at the conference, you may submit an abstract in Plain Text, PDF or MS Word formats to abstracts(a)scipy.org -- the deadline for abstract submission is July 6, 2007. Papers and/or presentation slides are acceptable and are due by August 3, 2007. Tutorial Sessions ----------------- Last year's conference saw an overwhelming turnout for our first-ever tutorial sessions. In order to better accommodate the community interest in tutorials, we've expanded them to 2 days and are providing food (requiring us to charge a modest fee for tutorials this year). A tentative list of topics for tutorials includes: - Wrapping Code with Python (extension module development) - Building Rich Scientific Applications with Python - Using SciPy for Statistical Analysis - Using SciPy for Signal Processing and Image Processing - Using Python as a Scientific IDE/Workbench - Others... This is a preliminary list; topics will change and be extended. If you'd like to present a tutorial, or are interested in a particular topic for a tutorial, please email the SciPy users mailing list (link below). A current list will be maintained here: http://www.scipy.org/SciPy2007/Tutorials Coding Sprints -------------- We've dedicated the Saturday after the conference for a Coding Sprint. Please include any ideas for Sprint topics on the Sprints wiki page here: http://www.scipy.org/SciPy2007/Sprints We're looking forward to another great conference! Best, Travis ------------- Links to various SciPy and NumPy mailing lists may be found here: http://www.scipy.org/Mailing_Lists

1 0

pickable ndarray subclass
by Christian K 17 Apr '07

17 Apr '07

Hi, could someone please provide example code for how to make a subclassed ndarray pickable? I don't quite understand the docs of ndarray.__reduce__. My subclassed ndarray has just one additional attribute. Thanks, Christian

2 2

Re: [Numpy-discussion] building numpy with Intel MS compiler and FFT
by Ray Schumacher 15 Apr '07

15 Apr '07

Thanks Rex, I'll give it a try next week. >I've compiled both Numpy and Python 2.5 with the Intel compiler. On a >Core 2 Duo, at least, the speed increase on Pybench was ~49%, even >before compiling Python with icc. My post about it was on 25 Jan, and >has subject: Compiling Python with icc

1 0

Re: [Numpy-discussion] newbie question - large dataset
by Steve Staneff 15 Apr '07

15 Apr '07

Many thanks to all those who responded to my question. Not being a real programmer, sometimes it's good to be reminded of the basics - such as timing various sections of code. It turns out that most of the time in the loop was spent on two of the functions. I've rewritten the first to be pre-calculated (a real oversight on my part), and looked at improving the second as much as possible. Some of the other time in the loop is being spent going back to the db server for data, and I'll look into using pytables as a way of speeding that up (possibly by initializing temporary tables on each of my client machines). Thanks again, Steve > On 4/7/07, Steve Staneff <staneff(a)constructiondatares.com> wrote: > >>Hi, >> >>I'm looking for a better solution to managing a very large calculation. >>Set A is composed of tuples a, each of the form a = [float, string]; set B >>is composed of tuples of similar structure (b = [float, string]). For >>each possible combination of a and b I'm calculating c, of the form c = >>f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial >>functions. >

1 0

ANN: PyQwt3D-0.1.4 released
by Gerard Vermeulen 15 Apr '07

15 Apr '07

What is PyQwt3D? - it is a set of Python bindings for the QwtPlot3D C++ class library which extends the Qt framework with widgets for 3D data visualization. PyQwt3D inherits the snappy feel from QwtPlot3D. The examples at http://pyqwt.sourceforge.net/pyqwt3d-examples.html show how easy it is to make a 3D plot and how to save a 3D plot to an image or an (E)PS/PDF file. - it requires and extends PyQt, a set of Python bindings for Qt. - it supports the use of PyQt, Qt, QwtPlot3D, and NumPy or SciPy in a GUI Python application or in an interactive Python session. - it runs on POSIX, Mac OS X and Windows platforms (practically any platform supported by Qt and Python). The home page of PyQwt3D is http://pyqwt.sourceforge.net. PyQwt3D-0.1.4 is a minor release providing: - support for SIP-4.6 - a binary installer for Windows. PyQwt3D-0.1.4 supports: 1. Python-2.5.x, -2.4.x, or -2.3.x 2. PyQt-4.2 and -4.1.x, or -3.17.x 3. SIP-4.6, or -4.5.x 4. Qt-4.2.x, Qt-4.1.x, Qt-3.3.x, or -3.2.x 5. QwtPlot3D-0.2.6 Enjoy -- Gerard Vermeulen

1 0