Mailman 3 January 2002 - NumPy-Discussion

numpy installation on HP-UX
by SungPil Yoon 31 Jan '02

31 Jan '02

Hi, all I have been trying to install numpy. Python is installed in my home directory ~/Python-2.2 and when I do % python setup.py install in the directory ~/Numeric-20.3, I get the following error message distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/local/lib/python2.2/config/Makefile (No such file or directory) Can anybody figure out what I am doing wrong? sungpil

1 0

Re: memory-mapped Numeric arrays: arrayfrombuffer version 2
by kragen＠pobox.com 29 Jan '02

29 Jan '02

Paul Dubois writes: > I have verified that this package seems to work on Windows. I says seems > only because I didn't try enough to uncover anything subtle. Thanks! If you run maparray.py from the command-line, it runs a basic regression test suite, which doesn't really try enough to uncover anything subtle either. > Unless or until we are convinced as a community that this is (a) the > right way to do this and (b) that the package is portable, it would not > be wise to put it in the main distribution. Well, I'm not the community, but I'll state my arguments. On (a): I don't know about the right way to do it; as I'm sure is obvious, I'm new to extending Numerical Python in C, but I doubt there's a simpler way to do it ("it" being seeing the contents of files as Numeric arrays), and I think it does work as well as it's possible to get it to work without major hacking of Numeric (to support read-only arrays) or the mmap module (to prevent closing an open object, to allow read-only mmapping on Windows). The other things on the wishlist are basically features to add --- "start" and "size" arguments to maparray(), a "create-a-file" argument to maparray(), etc., and don't change the basic structure. On (b): it depends on two-argument mmap.mmap(), open(), <file>.fileno(), and os.fstat(). The major portability hurdle there is probably mmap.mmap(), but that's OK. > I would like to hear from the community about this so that I will know > whether or not to add this package as a separate SourceForge 'package' > within the Numerical Python area. Meantime I will add a link to the web > page. I would too. There have been downloads from something like 50 IP addresses, but I've only heard from three people.

1 0

RandomArray and numarray
by Magnus Lie Hetland 29 Jan '02

29 Jan '02

Is there some support for random arrays in numarray? Or, if it becomes part of the standard library, will there be a separate module (such as Numeric's RandomArray)? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org

2 1

numarray bug...
by Magnus Lie Hetland 29 Jan '02

29 Jan '02

My numarray dumps core... >>> from numarray import * >>> a = zeros([10, 10]) >>> a = zeros([10, 10]) + 0.5 Bus Error (core dumped) Numeric tackles this just fine... What gives? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org

1 0

Re: What are the chances of getting numarray properties?
by Jay T Miller 29 Jan '02

29 Jan '02

Numarray should soon support the following properties: .shape .flat .real .imag .imaginary We're testing them here at STSCI. If all goes well, there should be a new numarray release within 1-3 weeks. Todd -- Todd Miller jmiller(a)stsci.edu STSCI / SSG (410) 338 4576

2 1

Accessors
by Magnus Lie Hetland 29 Jan '02

29 Jan '02

According to the (probably outdated) user guide for numarray[1] attributes have accessors such as a.foo() --> get foo a.foo(val) --> set foo to val but this doesn't seem to be the case. I see that the shape() method doesn't take any arguments; I guess the reshape method is the thing to use here. How about other attributes? Will they have special names for their setters too, or is this only true for the shape attribute? It seems to me that if we are going to abandon plain Python attributes (or 2.2 properties), it would be nice to handle the transition in a consistent manner... Perhaps? (But then again, I'm sure I'm missing something :) [1] http://stsdas.stsci.edu/numarray/Userguide.html -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org

1 0

What are the chances of getting numarray properties?
by Magnus Lie Hetland 29 Jan '02

29 Jan '02

I see in the docs that numarray uses accessor methods instead of properties, mainly because of performance (to avoid __getattr__ and __setattr__). With the new property type in Python 2.2, that shouldn't be as important anymore... Perhaps it would be possible to use properties where this type is available, and accessors elsewhere? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org

1 0

conditional array indexing
by Michaell Taylor 29 Jan '02

29 Jan '02

Very new user to Numpy, so excuse the question. It relates to a fundamental issue which I have having trouble getting a grip on. Specifically, say I have the following: a = ([1,2,1,2,4,3,1,2,3,2,3,2]) b = ([4,3,5,2,4,5,3,6,3,2,5,6]) c = random variable of length=len(b) I would like to get the value of C associated with the lowest value of b within groupings of a. That is, there are three incidents of "1" in the a array. The lowest value of b associated with the value 1 in the a index is "3". This occurs in index value 6 on the a and b arrays. Now, I would like to then know the value of c[6]. Obviously the real problem is far more complex with len(a)==5000. Any ideas? Thanks in advance. Michaell

3 3

amap: new function / ufunc method?
by John J. Lee 29 Jan '02

29 Jan '02

Having written all this down, I'm half-expecting somebody is going to point out some simple way of doing this with existing builtin functions, but here goes... In the process of 'porting' Konrad Hinsen's Histogram class (from the Scientific package) to N dimensions, I ran across the problem of how to apply a function to an array for a specified set of indices. In this case, the problem is to increment an N-dimensional array for each of a set of indices. In general, it occurred to me that it would be useful to have a function (named 'amap' here) to loop over a specified set of indices, applying a function to a set of arguments taken from a set of argument arrays (similar to the map buitin function in this respect), and filling in the corresponding elements in a target array with the results. For example, here is a bit of the histogram problem: [The original 1D Histogram class uses a clever array-broadcasting trick which creates intermediate rank-2 arrays, feeding in the data in chunks to avoid quadratic behaviour. This is fine with 1D arrays, but in poor taste for the N dimensional case, if it works at all.] #!/usr/bin/env python import Numeric as N def flatten_indices(shape, indices): dims = indices.shape[0] facs = pow(shape, N.arange(dims))[::-1,N.NewAxis] return N.add.reduce(facs*indices) def amap(target, function, indices, *arg_arrays): indices = N.asarray(indices) arg_arrays = map(N.asarray, arg_arrays) if not isinstance(target, N.ArrayType): raise TypeError, "target must be an array" if not target.iscontiguous(): raise ValueError, "target must be contiguous" if not N.alltrue([aa.iscontiguous() for aa in arg_arrays]): raise ValueError, "all arg_arrays must be contiguous" if not len(indices.shape) == 1: raise ValueError, "indices must be rank-1" tf = target.flat for index in indices: args = [arg_array.flat[index] for arg_array in arg_arrays] tf[index] = function(*args) # construct a 2 dimensional histogram given indices, ind, of data points nbins = (5, 4) # 5 bins along one axis, 4 along the other histo = N.zeros(nbins) # empty histogram # these are the indices into the histogram at which data points lie: # histo[1,0] should be incremented twice, histo[2,2] should be incremented # once, etc. ind = N.array([[1, 1, 2, 4], [0, 0, 2, 3]]) flat_ind = flatten_indices(histo.shape, ind) assert flat_ind == N.array([4, 4, 10, 19]) # Now we want to increment histo for each index pair ind[:,n] (in this # 2D case) def increment(x): return x+1 amap(histo, increment, flat_ind, histo) print histo Actually, I just realised that calling it something related to 'map' is rather misleading, because the builtin Python map generates a list of the same length as the sequence passed in; this function doesn't: it 'generates' (fills in, in fact) an array of the same size as the target passed in. Can't think of a better name, though. So: is there a way of doing this (implementing amap) efficiently already? I don't think there is, but I'd be interested to be proved wrong. I admit I don't like the relatively large number of arguments, but I still think it is worthwhile. You could make the *arg_arrays a single argument, and make target an optional argument, at the end of the arguments: def amap(function, indices, arg_arrays, target=None): so that by default, the target array is first created as zeros, then filled in and returned, or perhaps even better: def amap(function, arg_arrays, indices=None, target=None): though the special case where you don't give the indices argument is already achievable without this function, so it could be argued that hiding away the indices argument as a mere optional argument obscures the real point of the function. The last form also has the (debatable) benefit of being similar to the built-in Python map: def map(function, *args): The example call above then looks like this: histo = amap(increment, [histo], indices=flat_ind) Another couple of tweaks might be to allow scalar values to replace the arg_arrays argument, and to allow normal indices rather than flattened ones (the only reason I started out with flat indices was that put() did the same): def amap(function, args, indices=None, target=None): So that the example call looks like: histo = amap(increment, [histo], indices=ind) or histo = amap(Numeric.add, [histo, 1], ind) I like those last two. Even if there is already an efficient way of doing this with existing Numeric functions, I think this is a natural way of thinking about some operations on arrays -- this one, at least -- and would be useful to have in Numeric. For real efficiency, I suppose the right thing to do would be to have a map method on ufuncs, as is the case for reduce: histo = Numeric.add.map([histo, 1], ind) but it would be more work to implement than just a standard function, and less general. I've made a first attempt at the function, in C (will upload diff against 20.2.0 if anyone so requests). It only accepts one arg array, probably doesn't really do exactly the same as the Python function above (for better or worse), and it's a mess of a function, since I rarely write anything in C. It is also incorrect in at least some ways (most obviously, it only deals with Python floats / C doubles / Numpy PyArray_DOUBLEs), though there are some tests (which pass). I'm wasn't sure how fast it would be, since after all it's only using ordinary Python functions, not ufuncs, but my crude tests appeared to show an improvement of about a factor of two over an optimised version of the Python function above. The slow bit seems to be constructing the argument list / passing arg list to function, so I removed that to make the comparison, along with an unecessary attribute access in the loop. Presumably the general (> 1 arg_array) version would show a bigger improvement over the corresponding Python version. By the way, does anyone understand what the reduceat ufunc method is supposed to do? Seems to be the Beale cipher of Numeric -- having studied its output to see if it was the function I was looking for, I can find no sense in it and kind of suspect that it was constructed only to keep us puzzling. ;) Is Jim Hugunin not available for comment on the matter, or is this somebody else's dirty work? John

4 4

Odd numarray compile problem
by Magnus Lie Hetland 29 Jan '02

29 Jan '02

I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but somehow the compilation gets stuck on this line: gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray -I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c -o build/temp.linux-i686-2.2/_ufuncmodule.o No error message, no crash; it just never finishes -- it freezes. I compiled/installed it on a Solaris box earlier, with no problems whatsoever. I've never seen gcc freeze like this; can there be some circular dependencies or something? (This machine had some problems with its clock earlier, which made make go in a loop...) I tried running touch on all files, but that didn't change anything... Any ideas, or tips for finding out what's wrong? (BTW, Numeric compiles and installs just fine...) -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org

2 1