Mailman 3 March 2005 - NumPy-Discussion

Thoughts about zero dimensional arrays vs Python scalars
by Ralf Juengling March 20, 2005

March 20, 2005

Travis, Discussing zero dimensional arrays, the PEP says at one point: ... When ndarray is imported, it will alter the numeric table for python int, float, and complex to behave the same as array objects. Thus, in the proposed solution, 0-dim arrays would never be returned from calculation, but instead, the equivalent Python Array Scalar Type. Internally, these ArrayScalars can be quickly converted to 0-dim arrays when needed. Each scalar would also have a method to convert to a "standard" Python Type upon request (though this shouldn't be needed often). I'm not sure I understand this. Does it mean that, after having imported ndarray, "type(1)" to "ndarray.IntArrType" rather than "int"? If so, I think this is a dangerous idea. There is one important difference between zero dimensional arrays and Python scalar types, which is not discussed in the PEP: arrays are mutable, Python scalars are immutable. When Guido introduced in-place operators in Python, (+=, *=, etc.) he decided that "i += 1" should be allowed for Python scalars and should mean "i = i + 1". Here you have it, it means something different when i is a mutable zero dimensional array. So, I suspect a tacit re-definition of Python scalars on ndarray import will break some code out there (code, that does not deal with arrays at all). Facing this important difference between arrays and Python scalars, I'm also not sure anymore that advertising zero dimensional arrays as essentially the same as Python scalars is such a good idea. Perhaps it would be better not to try to inherit from Python's number types and all that. Perhaps it would be easier to just say that indexing an array always results in an array and that zero dimensional arrays can be converted into Python scalars. Period. Ralf PS: You wrote two questions about zero dimensional arrays vs Python scalars into the PEP. What are your plans for deciding these?

3 6

The first try on Numeric3.
by Daehyok Shin March 20, 2005

March 20, 2005

Dear Travis. I found no problem in installing Numeric3 and running tests in Mandrake 10.1. Good job. One question I have. I found you does not use UnitTest for test files. Will you change all tests eventually using UnitTest? If so, I think I can contribute something because I got some experience for it. For test files, please let me know what kind of supports you need. Thanks for your effort. -- Daehyok Shin Geography Department University of North Carolina-Chapel Hill USA

2 1

Re: [Numpy-discussion] Thoughts about zero dimensional arrays vs Python scalars
by Colin J. Williams March 20, 2005

March 20, 2005

Ralf Juengling wrote: >>> >>> >> It looks as though a decision has been made. I was among those who >> favoured abandoning rank-0 arrays, we lost. >> >> To my mind rank-0 arrays add complexity for little benefit and make >> explanation more difficult. > > > What the current PEP describes is perhaps close to what you want, > though: It says that indexing an array never results in a zero > dimensional array but it results "Array Scalars", which are basically > Python scalars, but there are just more of them to support variety > the numeric types. > > You could still create zero dimensional arrays by reshaping single > element arrays though. > >> >> I don't spot any discussion in the PEP of the pros and cons of the nd >> == 0 case. > > > I don't remember your idea--getting rid of zero dimensional arrays > altogether--being voiced and discussed on this list. What would be > the bad consequences of getting rid of zero dimensional arrays? The argument made in the PEP against returning Python scalars is: Cons: It is difficult to write generic code because scalars do not have the same methods and attributes as arrays. (such as .type or .shape). Also Python scalars have different numeric behavior as well. This results in a special-case checking that is not pleasant. Fundamentally it lets the user believe that somehow multidimensional homoegeneous arrays are something like Python lists (which except for Object arrays they are not). I suggest that, in striking the balance between the developer or generic writer and the end user, the greater design consideration should go to the ease and convenience of the end user. Colin W.

2 1

Additions to stdlib
by Colin J. Williams March 20, 2005

March 20, 2005

Here are some thoughts from Martin Löwis on the requirements and the process to enhance the standard Python library. Colin W. -------- Original Message -------- Subject: Re: survey of modules to be added to stdlib Date: Fri, 18 Mar 2005 23:26:31 +0100 From: "Martin v. Löwis" <martin(a)v.loewis.de> To: Alia Khouri <alia_khouri(a)yahoo.com> Newsgroups: comp.lang.python References: <1111184161.122375.227250(a)l41g2000cwc.googlegroups.com> Alia Khouri wrote: > BTW is there an official set of conditions that have to be met before a > module can be accepted into the stdlib? Yes - although this has never been followed to date: In PEP 2, http://www.python.org/peps/pep-0002.html a procedure is defined how new modules can be added. Essentially, we need a document stating its intended purpose, and a commitment by the authors to maintain the code. This may rule out inclusion of some modules in your list, e.g. if nobody steps forward to offer ongoing maintenance. Just that users want to see the code in the library is not sufficient, we also need somebody to do the actual work. If none of the core developers respond favourably to requests for inclusion, a library PEP can be seen as a last resort to trigger a BDFL pronouncement. Depending on the module, I personally would actively object inclusion if I have doubts whether the module is going to be properly maintained; I will, of course, obey to any BDFL pronouncement. Furthermore, and more recently, we also started requiring that code is *formally* contributed to the PSF, through the contrib forms, http://www.python.org/psf/contrib.html This may rule out further modules: the authors of the code have to agree to its inclusion in the library; somebody else contributing the modules for the authors will not be acceptable. However, the authors don't have to offer ongoing support for the copy in Python - any other volunteer could step in instead. Regards, Martin

1 0

A pray from an end user of numeric python.
by Daehyok Shin March 19, 2005

March 19, 2005

As an end user for Numeric and then numarray, recently I was quite frustrated by the move for Numeric3. Shortly after I became familiar with Numeric several years ago, I jumped to numarray mainly because of its more flexible indexing scheme. And after quite investment to learn the new package, recently I heard a distant echo that a new library called Numeric3 will replace Numeric sooner or later, and then I see a lot of discussion to discuss about better design of the new package in this mailing list. Because I think some feedbacks from end users are needed in the discussion, I dare to send this email, in spite of my meager knowledge about programming. To gods in numeric programming of Python, I want to make clear one basic fact. Like me, numerous scientific and engineering end users using Python needs a SINGLE standard data model for numeric operations. Even in the case that the data model has some defects in its design, many end users may not care about them, if many numeric packages using the data model are available. In this aspect, in my opinion, the replacement of current standard array types in Python must gain the top priority. How about building a very small array package with only MINIMUM set of functions first? We can think the functions of current array type in the standard Python library may define what is the minimal functions. All advanced mathematical or other functions may be added later as separate packages. So, I pray to gods in numeric programming of Python. Please give us a single numeric array model. Please save a flock of sheep like me from wondering around Numeric, numarray, Numeric3, or maybe in future Numeric4, 5, 6 …. Hopefully, this pray will get some attention. Thanks for reading this humble email. -- Daehyok Shin (Peter) Geography Department University of North Carolina-Chapel Hill USA

2 2

Status of numeric3 / scipylite / scipy_core
by Travis Oliphant March 18, 2005

March 18, 2005

I wanted to let people who may be waiting, that now is a good time to help with numeric3. The CVS version builds (although I'm sure there are still bugs), but more eyes could help me track them down. Currently, all that remains for the arrayobject is to implement the newly defined methods (really it's just a re-organization and re-inspection of the code in multiarraymodule.c to call it using methods). I also need to check the multidimensional slicing syntax when mixed with ellipses and slice objects so that take can be (functionally) replaced with multidimensional slicing. Any input on this would be appreciated. I'm referring to the fact that I think that a[...,ind,:] should be equivalent to take(a,ind,axis=-2). But, this necessitates some re-thinking about what partial indexing returns. What should a[:,ind1,:,ind2,:] return if a is a five-dimensional array? Currently, the proposed PEP for partial indexing always has the result as the broadcasted shape of ind1 and ind2 + the dimensionality of the un-indexed subspace. In otherwords, the unindexed subspace shape is always appended to the end of the result shape. I think this is wrong at least for the case of 1 indexing array because it does not let a[...,ind,:] be a replacement for take(a,ind,axis=-2). Is it wrong for more than 1 indexing array? To clarify the situation: Suppose X has shape (10,20,30,40,50) and suppose ind1 and ind2 are both broadcastable to the shape (2,3,4). Note for reference that take(X,ind1,axis=-2).shape returns (10,20,30,2,3,4,50) Now, according to the current proposal: X[..., ind1, :] will return a (2,3,4,10,20,30,50) --- I think this should be changed to return the same as take.... X[ind1, ind1, ind1, ind1, ind1] will return a (2,3,4) array (all dimensions are indexed) --- O.K. X[ind1, ind1, ind1, ind1] will return a (2,3,4,50) array X[ind1, ind1, :, ind1, ind1] will return a (2,3,4,30) array X[...,ind1,ind1,ind1] returns a (2,3,4,10,20) array --- is this right? X[:,ind1,:,ind2,:] returns a (2,3,4,10,30,50) array result[i,j,k,:,:,:] = X[:,ind1[i,j,k],:,ind2[i,j,k],:] So, here's the issue (if you are not familiar with the concept of subspace you can replace the word subspace with "shape tuple" in the following): - indexing with multidimensional index arrays under the numarray-introduced scheme (which seems reasonable to me) creates a single "global" subspace for all of the index arrays provided (i.e. there is no implied outer-product). - When there is a single index array it is unambiguous to replace the single-axis subspace with the index array subspace: i.e. X[...,ind1,:] can replace the second-to-last axis shape with the ind1.shape to get a (10,20,30,2,3,4,50) array. - Where there is more than one index array, what should replace the single-axis subspaces that the indexes are referencing? Remember, all of the single-axis subspaces are being replaced with one "global" subspace. The current proposal states that this indexing subspace should be placed first and the "remaining subspaces" pasted in at the end. Is this acceptable, or can someone see a problem?? Best regards, -Travis

6 13

Re: Trying out Numeric3
by Michiel Jan Laurens de Hoon March 18, 2005

March 18, 2005

Travis Oliphant wrote: > Michiel Jan Laurens de Hoon wrote: >> During compilation, I noticed that some test program is run, >> presumably for configuration. The test program is compiled by a >> different compiler as the one used in the build process. Note that >> "python setup.py config" is available in the standard distutils, so it >> may be better to use that instead of a self-defined configuration >> tool. For one thing, it'll make sure that the compiler used for >> configuration is the same as the one used for compilation. > > What does python setup.py config do? I have been unable to figure out > how to get the configuration I need. If you have suggestions, that > would be great... > I submitted a patch to sourceforge that modifies setup.py such that it uses distutils' stuff for the configuration. See patch #1165840. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

2 1

Re: Please chime in on proposed methods for arrays
by Joe Harrington March 18, 2005

March 18, 2005

I'll start by saying something positive: I am very encouraged by all the work that's going into resolving the small/big array split! That said, I view a.Sin as a potentially devastating change, if traditional functional notation is not guarranteed to be preserved forever. >From a learning user's perspective, having to say a.sin() or a.Sin rather than sin(a) will be enough to make most people stay away from numerical python. I say this from experience: most of what Python does well, guile also did well. After the IDAE BoF at the 1996 ADASS meeting, we considered whether guile would be a better platform than Python. It had a lot of development force behind it, had all the needed features, etc. We asked our audience whether they would use a lisp-based language. There was laughter. The CS people here know that lisp is a "beautiful" language. But nobody uses it, because the syntax is so different from the normal flow of human thought. It is written to make writing an interpreter for it easy, not to be easy to learn and use. I've tried it about 8 times and have given up. Apparently others agree, as guile has been ripped out of many applications, such as Gimp, or at least augmented as an extension language by perl or python, which now get most of the new code. Normal people don't want to warp their brains in order to code. Consider the following: math: 2 1/2 -b +- (b - 4ac) x = ------------------ 2a IDL: x = (-b + [1.,-1] * sqrt(b*b-4*a*c)) / (2*a) lisp: (let x (/ (+ (- b) (sqrt (- (* b b) (* 4 a c)))) (* 2 a))) You can verify that you have coded the IDL correctly at a glance. The lisp takes longer, even if you're a good lisp programmer. Now consider the following common astronomical equation: sin(dec) - sin(alt) sin(lat) cos(a) = ---------------------------- cos(alt) cos(lat) IDL: a = acos((sin(dec) - sin(alt) * sin(lat)) / (cos(alt)*·cos(lat))) proposal: a = ((dec.Sin - alt.Sin * lat.Sin) / (alt.Cos * lat.Cos)).Acos readable, but we start to see the problem with the moved .Acos. Now try this: 2 sin x + cos(tan(x + sin(x))) a = e a = exp((sin(x))^2 + cos(tan(x + sin(x)))) a = (x.Sin**2 + (x + x.Sin).tan.cos).Exp Half of it you read from left to right. The other half from right to left. Again, the IDL is much easier to write and to read, given that we started from traditional math notation. In the proposal version, it's easy to overlook that this is an exponential. So, I don't object to making functions into methods, but if there's even a hint of deprecating the traditional functional notation, that will relegate us to oblivion. If you don't believe it still, take the last equation to a few non-CS types and ask them whether they would consider using a language that required coding math in the proposed manner versus in the standard manner. Then consider how much time it would take to port your existing code to this new syntax, and verify that you didn't misplace a paren or sign along the way. A statement that traditional functional notation is guarranteed always to be part of Numeric should be in the PEP. Even calling it syntactic sugar is dangerous. It is the fundamental thing, and the methods are sugar for the CS types out there. --jh--

7 7

[Fwd: Re: [Numpy-discussion] Status of numeric3 / scipylite / scipy_core]
by Travis Oliphant March 17, 2005

March 17, 2005

I originally sent the attached just to Tim. It was meant for the entire list.

1 0

casting in numarray
by Joachim Boomberschloss March 17, 2005

March 17, 2005

Hi, I'm using numarray for an audio-related application as a buffer in an audio-processing pipeline. I would like to be able to allocate the buffer in advance and later regard it as a buffer of 8bit or 16bit samples as appropriate, but in numarray, casting always produces a new array, which I don't want. How difficult should it be to make it possible to create an array using an exsisting pre-allocated buffer to act as an interface to that buffer? Also, if others consider it useful, is there anyone willing to guide me through the code in doing so? Thanks, Joe __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

2 1