Mailman 3 March 2007 - NumPy-Discussion

Adapting algorithm to accept Scipy sparse matrix
by David Koch March 26, 2007

March 26, 2007

Hi, I implemented an algorithm in NumPy which assumes that the input is of type ndarray, so elementwise multiplication is done as dot(x,y), equation solving using linalg.solve etc. I now want to modify the whole thing to accept scipy.sparse matrices (which for instance has linsolve.spsolve instead of linalg.solve) and I am wondering what's the best way to go about it. I don't want to check sparse.issparse all the time or even create a separate sub-class for each data type ... what to do, Ahh! Help! /David

2 5

Modular toolkit for Data Processing 2.1 released!
by Tiziano Zito March 26, 2007

March 26, 2007

MDP version 2.1 and symeig 1.2 have been released! What's new in version 2.1? -------------------------- - Fully compatible with NumpPy 1.0, the first stable release of the descendant of the Numeric python extension module - symeig project resumed and updated - For increased speed, scipy and symeig are automatically used if available - New nodes: Independent Slow Feature Analysis and quadratic forms analysis algorithms - General improvements, several bug fixes, and code cleanups What is it? ----------- Modular toolkit for Data Processing (MDP) is a data processing framework written in Python. From the user's perspective, MDP consists of a collection of trainable supervised and unsupervised algorithms that can be combined into data processing flows. The base of readily available algorithms includes Principal Component Analysis, two flavors of Independent Component Analysis, Slow Feature Analysis, Independent Slow Feature Analysis, and many more. From the developer's perspective, MDP is a framework to make the implementation of new algorithms easier. MDP takes care of tedious tasks like numerical type and dimensionality checking, leaving the developer free to concentrate on the implementation of the training and execution phases. The new elements then seamlessly integrate with the rest of the library. MDP has been written in the context of theoretical research in neuroscience, but it has been designed to be helpful in any context where trainable data processing algorithms are used. Its simplicity on the user side together with the reusability of the implemented nodes make it also a valid educational tool. As its user base is increasing, MDP is becoming a common repository of user-supplied, freely available, Python-implemented data processing algorithms. The optional symeig module contains a Python wrapper for the LAPACK functions to solve the standard and generalized eigenvalue problems for symmetric (hermitian) positive definite matrices. Those specialized algorithms give an important speed-up with respect to the generic LAPACK eigenvalue problem solver used by NumPy. Resources --------- Download: http://sourceforge.net/project/showfiles.php?group_id=116959 Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://sourceforge.net/mail/?group_id=116959 -- Tiziano Zito Institute for Theoretical Biology Humboldt-Universitaet zu Berlin Invalidenstrasse, 43 10115 Berlin, Germany Pietro Berkes Gatsby Computational Neuroscience Unit Alexandra House, 17 Queen Square London WC1N 3AR, United Kingdom

2 1

random state not portable?
by Pietro Berkes March 26, 2007

March 26, 2007

Dear Numpy devels, First of all, thank you for the great job done so far! We've been updating our MDP module, and had a chance to appreciate it. We found an issue in the random module: the random state contains an array of int32 or int64 depending on the architecture, making it not portable. For example, try the following on two machines with different architecures: Python 2.4.4 (#2, Jan 13 2007, 17:50:26) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import platform >>> import pickle >>> >>> platform.architecture() ('32bit', '') >>> >>> state = numpy.random.get_state() >>> pickle.dump(state, file('/tmp/random_state.pic','w')) >>> numpy.random.permutation(numpy.arange(10)) array([8, 6, 3, 1, 0, 5, 7, 4, 2, 9]) >>> Python 2.5 (r25:51908, Mar 19 2007, 13:41:07) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import platform >>> import pickle >>> >>> platform.architecture() ('64bit', 'ELF') >>> >>> state = pickle.load(file('/tmp/random_state.pic','r')) >>> numpy.random.set_state(state) >>> numpy.random.permutation(numpy.arange(10)) array([3, 9, 4, 8, 1, 6, 2, 0, 5, 7]) >>> Is there any known workaround? Thanks! -- Tiziano Zito Institute for Theoretical Biology Humboldt-Universitaet zu Berlin Invalidenstrasse, 43 10115 Berlin, Germany Pietro Berkes Gatsby Computational Neuroscience Unit Alexandra House, 17 Queen Square London WC1N 3AR, United Kingdom

1 0

Re: [Numpy-discussion] dtype confusion
by Jan Strube March 26, 2007

March 26, 2007

I'm afraid I'm not quite done with this, yet. There's still more I am confused about, even after reading the manual (again). There is something fundamentally weird about record arrays, that doesn't seem to click with me. Please have a look at this piece of code: import numpy as N newtype = N.dtype([('x', N.float64), ('y', N.float64), ('z', N.float64)]) x = N.zeros((10, 3)) x.dtype = newtype print x x = N.zeros((10, 3), dtype=newtype) print x I think I understand what's going on here: x = N.zeros((10, 1), dtype=newtype) print x So can I summarize that in the first case a (10,3) array is reinterpreted ( numpy.view) as a record array, while in the second case I am actually requesting a (10,3) array of the dtype, which itself has 3 elements? And the record array should not be viewed as a column-wise array with named columns (although it can be accessed that way), but rather as an array that groups the columns in each row into a record. Because after turning it into a record array, x is still c-contiguous and not fortran-contiguous... Maybe I'm actually starting to understand this a bit. Anyways, thanks for listening ;-) Cheers, Jan

1 0

subversion site down
by Christopher Hanley March 25, 2007

March 25, 2007

Hi, It appears that the subversion server is down for numpy. Chris

2 1

Simple multi-arg wrapper for dot()
by Bill Baxter March 25, 2007

March 25, 2007

I mentioned in another thread Travis started on the scipy list that I would find it useful if there were a function like dot() that could multiply more than just two things. Here's a sample implementation called 'mdot'. mdot(a,b,c,d) ==> dot(dot(dot(a,b),c),d) mdot(a,(b,c),d) ==> dot(dot(a,dot(b,c),d) mdot(a,(b,(c,d))) ==> dot(a,dot(b,dot(c,d)) --- def mdot(*args): """Multiply all the arguments using matrix product rules. The output is equivalent to multiplying the arguments one by one from left to right using dot(). Precedence can be controlled by creating tuples of arguments, for instance mdot(a,((b,c),d)) multiplies a (a*((b*c)*d)). Note that this means the output of dot(a,b) and mdot(a,b) will differ if a or b is a pure tuple of numbers. """ if len(args)==1: return args[0] elif len(args)==2: return _mdot_r(args[0],args[1]) else: return _mdot_r(args[:-1],args[-1]) def _mdot_r(a,b): """Recursive helper for mdot""" if type(a)==types.TupleType: if len(a)>1: a = mdot(*a) else: a = a[0] if type(b)==types.TupleType: if len(b)>1: b = mdot(*b) else: b = b[0] return numpy.dot(a,b) --- It does about twice as many function calls as just using dot which I'm guessing is mostly why its a good bit slower than just doing the nested dot calls. Maybe one of you smart folks can figure out a way to bring it closer to dot's speed. --bb

8 22

Re: [Numpy-discussion] dtype confusion
by Jan Strube March 25, 2007

March 25, 2007

Thanks Stefan and Travis for their explanations. First a request: Could both explanations be added to the manual, please? Thanks. So the problem I was having was that I thought this difference in behavior would be caused by two different types: recarray and ndarray. I feel that there is still some underlying behavior that I don't understand, but for now the puzzling thing was that I thought that the new dtype simply assigns names and aliases varname[:,0] to varname['x'], varname[:,1] to varname['y'], and so on. I didn't expect it to fundamentally change the array in that way, I would have expected varname.view(recarray) to do that. Would it make sense for me then to create something like a "namedarray" subclass of ndarray that doesn't change the dtype but simply let's you provide names and assign them to the columns? Maybe in my case it's easier, because all columns of the record array have the same type I guess I'll re-read the dtypes and recarray parts of the manual now... Thanks, Jan

1 0

New Operators in Python
by Travis Oliphant March 25, 2007

March 25, 2007

Every so often the idea of new operators comes up because of the need to do both "matrix-multiplication" and element-by-element multiplication. I think this is one area where the current Python approach is not as nice because we have a limited set of operators to work with. One thing I wonder is if we are being vocal enough with the Python 3000 crowd to try and get additional operators into the language itself. What if we could get a few new operators into the language to help us. If we don't ask for it, it certainly won't happen. My experience is that the difficulty of using the '*' operator for both matrix multiplication and element-by-element multiplication depending on the class of the object is not especially robust. It makes it harder to write generic code, and we still haven't gotten everything completely right. It is somewhat workable as it stands, but I think it would be nicer if we could have some "meta" operator that allowed an alternative definition of major operators. Something like @* for example (just picking a character that is already used for decorators). I wonder if we should propose such a thing for Python 3000. -Travis

10 9

array multiplication with different dtypes
by Jan Strube March 25, 2007

March 25, 2007

I'm having a difficult time understanding the following behavior: import numpy as N # create a new array 4 rows, 3 columns x = N.random.random((4, 3)) # elementwise multiplication x*x newtype = N.dtype([('x', N.float64), ('y', N.float64), ('z', N.float64)]) # interpret the array as an array of cartesian coordinates x.dtype = newtype x*x --> TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and ' numpy.ndarray' N.__version__ '1.0.2.dev3498' So just by assigning names to the columns, I can't multiply the arrays any more? Numpy itself still claims that x is an ndarray. I hope this is a bug, because it would be really convenient if this still worked after assigning names to columns. What I really wanted to do was N.dot(x, x.T) to get the length of the cartesian vectors, but (I think) this fails for the same reason. Please tell me this is a bug ;-) Cheers, Jan

2 1

Detect subclass of ndarray
by Charles R Harris March 25, 2007

March 25, 2007

Anyone, What is the easiest way to detect in python/C if an object is a subclass of ndarray? Chuck

6 25