Mailman 3 October 2004 - NumPy-Discussion

matrixmultiply: return type
by Alan G Isaac 16 Oct '04

16 Oct '04

Being new to numerical Python applications, I was a little puzzled/concerned when I read http://sourceforge.net/tracker/index.php?func=detail&aid=984368&group_id=13… I *think* the answer is: matrixmultiply will always return an array. Is there a stable view about what type of object will be returned by matrixmultiply? Currently, to my initial surprise, it returns an array when the arguments are matrices. Is this stable? Might an optional argument to specify the return type be desirable? Thank you, Alan Isaac

1 0

how to deal with large arrays
by Darren Dale 15 Oct '04

15 Oct '04

Hello, I have two 2D arrays, Q is 3-by-q and R is r-by-3. At each q, I need to sum q(r) over R, so I take a dot product RQ and then sum along one axis to get a 1-by-q result. I'm doing this with dot products because it is much faster than the equivalent for or while loop. The intermediate r-by-q array can get very large though (200MB in my case), so I was wondering if there is a better way to go about it? If not, I can slice up R and deal with it one chunk at a time, then the intermediate arrays fit within the available system resources. Would somebody offer a suggestion of how to do this intelligently? Should the intermediate array be about the size of the processor cache, some fraction of the available memory, or is there something else I need to consider? Thank you, Darren

2 2

New Release of LAPACK and ScaLAPACK Planned
by Jack Dongarra 15 Oct '04

15 Oct '04

New Release of LAPACK and ScaLAPACK planned. We are pleased to announce that we recently received NSF funding for new releases of the LAPACK and ScaLAPACK linear algebra libraries. The proposal pointed out the new and better algorithms that have been developed by many people in the community since the first releases of these libraries, as well as more obvious gaps and possible improvements. The proposal listed a large number of activities, which we now need to prioritize. There are a number of design decisions that still need to be made, for which we are interested in your input. For this purpose, we would like to remind you of a web page to collect your input that we originally announced on NA-Digest while we were preparing the proposal: http://icl.cs.utk.edu/lapack-survey.html In addition to the questions on that form, we are interested in your opinion on all aspects of the proposal, a copy of which you may find at: http://www.cs.berkeley.edu/~demmel/Sca-LAPACK-Proposal.pdf Thanks, Jim Demmel and Jack Dongarra --=20 Stephen Walton <stephen.walton(a)csun.edu> Dept. of Physics & Astronomy, CSU Northridge --=-vf5K3It096b9Vx529EKP Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQBBcw7pURWByv7S9xcRAms0AJ0YE13AXJ127J/5UVRs2t+BUYMIUQCgnd8I kvjNlPBX6phVfhjclKGExPY= =1kTj -----END PGP SIGNATURE----- --=-vf5K3It096b9Vx529EKP--

1 0

New Release of LAPACK and ScaLAPACK Planned
by Jack Dongarra 15 Oct '04

15 Oct '04

New Release of LAPACK and ScaLAPACK planned. We are pleased to announce that we recently received NSF funding for new releases of the LAPACK and ScaLAPACK linear algebra libraries. The proposal pointed out the new and better algorithms that have been developed by many people in the community since the first releases of these libraries, as well as more obvious gaps and possible improvements. The proposal listed a large number of activities, which we now need to prioritize. There are a number of design decisions that still need to be made, for which we are interested in your input. For this purpose, we would like to remind you of a web page to collect your input that we originally announced on NA-Digest while we were preparing the proposal: http://icl.cs.utk.edu/lapack-survey.html In addition to the questions on that form, we are interested in your opinion on all aspects of the proposal, a copy of which you may find at: http://www.cs.berkeley.edu/~demmel/Sca-LAPACK-Proposal.pdf Thanks, Jim Demmel and Jack Dongarra --=20 Stephen Walton <stephen.walton(a)csun.edu> Dept. of Physics & Astronomy, CSU Northridge --=-vf5K3It096b9Vx529EKP Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQBBcw7pURWByv7S9xcRAms0AJ0YE13AXJ127J/5UVRs2t+BUYMIUQCgnd8I kvjNlPBX6phVfhjclKGExPY= =1kTj -----END PGP SIGNATURE----- --=-vf5K3It096b9Vx529EKP--

1 0

Slow operation of nd_image.generic_filter
by akulla＠comcast.net 13 Oct '04

13 Oct '04

Hi all, Could it be that the execution of the following function lasts more than 25 seconds, for an array of shape (256, 480)? ... def myFunc(anArray, winSize=5): return numarray.nd_image.generic_filter(\ input=anArray, function=lambda win: win.mean(), size=winSize, mode='constant') ... Python 2.3, numarray 1.0 (XP, P4) Regards, Alban

1 0

Re: [Numpy-discussion] character arrays supported by C API?
by Todd Miller 13 Oct '04

13 Oct '04

On Thu, 2004-10-14 at 04:20, Faheem Mitha wrote: > On Wed, 15 Sep 2004, Todd Miller wrote: > > > On Wed, 2004-09-15 at 00:52, Faheem Mitha wrote: > >> Dear People, > >> > >> Are character arrays supported by the Numarray C API? My impression from > >> the documentation is no, but I would appreciate a confirmation. Thanks. > >> > >> Faheem. > > > > Yes and no. CharArray is not as well supported from C as NumArray; > > there are no easy to call functions which will convert a nested sequence > > of strings into a CharArray. > > > > However, it is possible to call the Python functions in the CharArray > > module from C, and a pre-existing CharArray is a PyArrayObject so it > > can be manipulated in C as a struct; it's shape and strides are > > visible, it's itemsize is the length of the string, etc. > > > > What is it you want to do? What functions do you think would help? > > Hi. Sorry about the slow reply. > > What I want to do is extremely simple. I want to convert (in C++) a C++ > character array to a CharArray. The simplest way of doing this would be to > create an array of the appropriate size, and write character strings into > it element by element. > > So, a utility function which creates a character array of appropriate > dimensions would be useful. Also a utility function which convert a list > of strings into a Character Array would also be desirable. > > Currently I am having to work around this limitation by returning lists of > strings back to Python. I'd prefer to not have to do that. That's a sensible addition, but right now, such a function does not exist, and I don't have time to add it myself. The way to achieve this without C-API support by CharArray is to do a Python callback. The steps in C would be roughly: 0. Import the numarray.strings module. PyImport_ImportModule(). 1. Get the module's dictionary object. PyModule_GetDict(). 2. Get a pointer to CharArray by looking it up in the dictionary. PyDict_GetItemString(). 3. Construct an argument tuple which contains the constructor parameters. Py_BuildValue(). 4. Call the constructor using the arg tuple. The return value is the CharArray. PyObject_CallFunction(). Similar steps are done for NumArray in the current C-API in newarray.ch in NA_NewAllFromBuffer(). Regards, Todd

1 0

A disconnected numarray rant
by Alexander Schmolck 13 Oct '04

13 Oct '04

Hi, I'm taking a 1 month break from computers (i.e. I will be completely off-line), and I have to catch a train in an hour; but I've recently bitten the bullet and made a matrix class I've been using for some time work with numarray; I've written down a number of things that occured to me while I was doing it, including some things which I think are bugs in numarray, so I thought at least posting the bugs would be a useful service; the rest is very raw and essentially unedited cut-and-paste of these notes -- sorry about that and I hope it doesn't contain anything particularly offensive. P.S. just dumped the code for the matrix class (nummat) at http://www.dcs.ex.ac.uk/~aschmolc/Stuff/ 'as The following are my notes: Things that fairly clearly seem to be bugs: - numarray.Int32 etc. can't be pickled - ``a = array(1+0j); a.imag = a.real * 10`` => IndexError - array(0, type=Float64) + 1e3000 => `inf` with right error modes but array(0, type=Float32) + 1e3000 => `OverflowError` - numarray.array(10)/numarray.array(0) => 0 - numarray.array(10000000000000L) => array(1316134912) - numarray.where(0,1,0) => array([0]) - l = [1,2,3]; numarray.put(l,numarray.array([1,2,0]),[0,0,0]); l => [1, 2, 3] a = array([1,2,3]); numarray.put(a,numarray.array([1,2,0]),[0,0,0]); a => array([0, 0, 0]) - repr(numarray.array([],typecode='i')) (etc. etc.) => "numarray.array([])" - getattr(array([1,2,3]), '_aligned') => SystemError - obscure: numarray.where(0, matrix(568, convert_scalars=True),2) => ValueError (tries __len__ which fails, as len(array(568)) also fails) Numeric incompatiblilities (that are either undocumented or bug-like) - numarray.array('a', typecode='O') => TypeError (object arrays) - for extra fun try: numarray.array(1, type=numarray.Object) -=> RuntimeError something entirely different - nonzero is completely incompatible - shape(None) etc. no longer works (IMHO a bug) - cross_correlate & average missing - left_shift et al missing - numarray.sqrt(a,a) is None (*not* the result, as it used to be) - num.put(a, [0,1,2,3], [10,20]) style behavior seems unavailable (without numarray.numeric) put(array([[ 0., 1., 2.], [ 3., 4., 5.]]), [1, 4], [10,40]) fails - boolean testing (not even bool(array(0)) works; I'm not sure this is good) - Generally different handling of rank0-arrays; e.g. ``type(num.array(1.0) + 0) is float``; one potentially very nasty gotcha are inplace operations (e.g. a**=2) which have totally different semantics for python scalars and rank0 arrays, which, unlike Attribute errors on ``a.shape``, can lead to nasty bugs in corner cases (e.g. when a reduction just infrequently yields scalar ``a``) -- I think this should be mentioned in a gotchas section (another possible entry would be the need to use .copy() to **save** memory on slicing and 1xN, Nx1 matrices versus vectors (people are not used to thinking properly about rank from mathematical training or matlab exposure)). - asarray downcasts arrays (e.g.: asarray(array([1.,2.,3.]),'i')) - numarray.ones(-5) => MemoryError (ValueError would be nicer) - numarray.ones(2.0), numarray.ones([2]) fail (cf. numarray.range(2.0)) b=num.array([[1,2,3,4],[5,6,7,8]]*2) assert eq(num.diagonal(b), [1,6,3,8]) assert eq(num.diagonal(b, -1), [5,2,7]) c = num.array([b,b]) assert eq(num.diagonal(c,1), [[2,7,4], [2,7,4]]) - no a.toscalar() !!! - matrixmultiply in the docs - what's the point of swapaxes (i.e. why not have a generalized in-place transpose?) - what's the point of innerproduct? - indexing by a list is different from indexing by tuple (I haven't had time to look closely at the docs whether that's intentional) - doesn't know about Numeric's bizzarre '\x0b' typecode - numarray.sqrt.reduce([]) raises (sensibly) TypeError, not ValueError - len(array(1)) or array(1)[0] won't work anymore (understandable, but should be documented) - (should maximim, minimum reduce to -inf and inf?) - <built-in method reduce of _BinaryUFunc object at 0x82dfc9c> is not a very helpful repr; should be possible to get to the ufunc itself - as in Numeric numarray.maximum.reduce(numarray.array([0,-0.])) => -0.0 - __array__ protocol no longer supported (how can a non-derived class convert itself efficiently to an array?) Documentation Gotchas - p. 34 IMO row vector is used incorrectly; row and column vectors are really matrices (i.e. have rank 2) so ``array([[1,2,3]])`` would be a row vector - No proper explanation of differences between Numeric and numarray, or numarray.numeric module differences to proper (e.g. argmin) - No migration and best-practice advice (e.g. there should be a standard way for packages which work with both numarray and numeric as backends to let the user choose his preference; how about setting an environment var NumPy or something?) Waffle ------ - there *really* ought to be an array equality function (with optional tolerance); it's quite difficult to get right for are normal user (nans; zero-size arrays etc.) and it's often required, especially for testing - rank preserving reduction seems useful as an option would be nice -- e.g. to subtract out or divide by the reduced portion (which currently won't e.g. work for columns without adding a unit-dimension by hand). Design The (AFAICS) benefit-free but downside-rich introduction of `type` '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Is there any reason that Typecode objects that compare as desired to the relevant strings ("i", "d") wouldn't have done? Now there is an explosion and confusion of interfaces -- some numpy code will now only except type(code)s as "typecode" keyword parameter (even in numarray! see numarray.mlab!) and other stuff Never mind that type already is a highly overused word in the python world. The big method bloat. ''''''''''''''''''''' As it says in the Numeric manual introductions there were "good reasons" for "very few array methods" -- now there are **56** public methods and 8 public attributes (public == not starting with '_'); of those 56 methods about 11 are accessors and of the rest about half are redundant or worse (i.e. they either also exist as numarray functions (argmin, argmax, diagonal, ...) or they really ought to be functions (mean, stddev) or they are quite confusing (``a.min``, ``a.max`` which behave quite differenlty from ``a.argmin`` and ``a.argmax``, never mind ``numarray.minimum``) or simply utterly pointless (``a.nelements`` == ``a.size``)). - argmin, argmax : what's wrong with numarray.argmin, numarray.argmax??? Why do argmin/argmax and max/min have completely different interfaces??? If there really is a need for these (there isn't) anything a.min and a.max should be called a.flatmin, a.flatmax - diagonal, mean, nelements, nonzero, ... - perversely the **only** function that I can think off that could have sensibly become a method hasn't: ``put`` (it used to work only on arrays under Numeric and not without reason, so making it a method would have been sensible; numarray.put of course also "works" on non-arrays, it just doesn't do anything with them) Test Code ''''''''' numtest.py doesn't inspire full confidence (it's about 1000 lines of actual code but it doesn't seem that clearly structured and AFAICT contains no single loop (and that despite the diversity of shapes, types etc. that exist in numarray -- why not try something slightly more systematic?)).

2 1

ANN: SciPy 0.3.2 Released
by Pearu Peterson 12 Oct '04

12 Oct '04

Hi, Scipy 0.3.2 has been released and binaries are available from the scipy.org site: http://www.scipy.org Scipy 0.3.2 is a bug fix release of Scipy 0.3 including the following new features: - wxPython 2.5 support - reading/writing dense/sparse matrices in Matrix Market format - iterative solvers, new functions sqrtm, hessenberg - Constrained Optimization BY Linear Approximation - discrete Boltzmann, Planck, Levy distributions - Scipy tests pass now also on 64-bit systems and Mac OSX etc. The complete release notes can be found here: http://www.scipy.org/download/scipy_release_notes_0.3.2.html Best regards, Pearu BTW Scipy is: ------------- Scipy is an open source library of scientific tools for Python. Scipy supplements the popular Numeric module, gathering a variety of high level science and engineering modules together as a single package. Scipy includes modules for graphics and plotting, optimization, integration, special functions, signal and image processing, genetic algorithms, ODE solvers, and others.

1 0

Cheap software for you please. 6610536
by Shelia Mendez 11 Oct '04

11 Oct '04

1 0

ieeespecial
by Darren Dale 10 Oct '04

10 Oct '04

Hello, I am getting invalid numeric result exceptions when dividing a complex array by zero. Is this the desired behavior? Also, while trying to find a way around the above problem, I ran ieeespecial.test and got the following output. I am running numarray 1.1 on python 2.3.3. Todd, this might be correlated with the numerix package in matplotlib. I tried importing numarray and ieeespecial without matplotlib and the ieeespecial.test was successful. Thanks, Darren In [31]: ieeespecial.test() Out[31]: inf ***************************************************************** Failure in example: inf # the repr() of inf may vary from platform to platform from line #6 of numarray.ieeespecial Expected: inf Got: Out[31]: nan ***************************************************************** Failure in example: nan # the repr() of nan may vary from platform to platform from line #8 of numarray.ieeespecial Expected: nan Got: Out[31]: (array([0, 2]), array([0, 3])) ***************************************************************** Failure in example: getinf(b) from line #20 of numarray.ieeespecial Expected: (array([0, 2]), array([0, 3])) Got: Out[31]: array([[ 999., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 999.], [ 12., 13., 14., 15.]]) ***************************************************************** Failure in example: a from line #26 of numarray.ieeespecial Expected: array([[ 999., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 999.], [ 12., 13., 14., 15.]]) Got: Out[31]: (array([0, 1, 2]), array([1, 2, 3])) ***************************************************************** Failure in example: getnan(a) from line #35 of numarray.ieeespecial Expected: (array([0, 1, 2]), array([1, 2, 3])) Got: ***************************************************************** 1 items had failures: 5 of 11 in numarray.ieeespecial ***Test Failed*** 5 failures. Out[31]: (5, 11) -- Darren

2 5

NumPy-Discussion October 2004