Mailman 3 May 2009 - NumPy-Discussion

Problem with correlate
by rob steed 18 May '09

18 May '09

Hi all, I have been using numpy.correlate and was finding something weird. I now think that there might be a bug. Correlations should be order dependent eg. correlate(x,y) != correlate(y,x) in general (whereas convolutions are symmetric) >>> import numpy as N >>> x = N.array([1,0,0]) >>> y = N.array([0,0,1]) >>> N.correlate(x,y,'full') array([1, 0, 0, 0, 0]) >>> N.correlate(y,x,'full') array([0, 0, 0, 0, 1]) This works fine. However, if the arrays have different lengths, we get a problem. >>> y2=N.array([0,0,0,1]) >>> N.correlate(x,y2,'full') array([0, 0, 0, 0, 0, 1]) >>> N.correlate(y2,x,'full') array([0, 0, 0, 0, 0, 1]) I believe that somewhere in the code, the arrays are re-ordered by their length. Initially I thought that this was because correlate was deriving from convolution but looking at numpy.core, I can see that in fact convolution derives from correlate. After that, it becomes C code which I haven't managed to look at yet. Am I correct, is this a bug? regards Rob Steed

3 2

Savetxt usage question
by Gökhan SEVER 17 May '09

17 May '09

Hello, Is there a way to write a header information to a text file using savetxt command besides dumping arrays in the same file? In little detailed fashion: I have to write a few long column of arrays into a text file. While doing that I need to put some information regarding to the context of the file. Like variable names, project date, missing value equivalent etc... So far, I couldn't see that this could be achieved with one savetxt command. However there might be an easy point that I am missing. Thank you. Gökhan

2 2

numpy slices limited to 32 bit values?
by Glenn Tarbox, PhD 17 May '09

17 May '09

I'm using the latest version of Sage (3.4.2) which is python 2.5 and numpy something or other (I will do more digging presently) I'm able to map large files and access all the elements unless I'm using slices so, for example: fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64', mode='r+', shape=(10000000000,)) which is 1e10 doubles if you don't wanna count the zeros gives full access to a 75 GB memory image But when I do: fp[:] = 1.0 np.sum(fp) I get 1410065408.0 as the result Interestingly, I can do: fp[9999999999] = 3.0 and get the proper result stored and can read it back. So, it appears to me that slicing is limited to 32 bit values Trying to push it a bit, I tried making my own slice myslice = slice(1410065408, 9999999999) and using it like fp[myslice]=1.0 but it returns immediately having changed nothing. The slice creation "appears" to work in that I can get the values back out and all... but inside numpy it seems to get thrown out. My guess is that internally the python slice in 2.5 is 32 bit even on my 64 bit version of python / numpy. The good news is that it looks like the hard stuff (i.e. very large mmaped files) work... but slicing is, for some reason, limited to 32 bits. Am I missing something? -glenn -- Glenn H. Tarbox, PhD || 206-274-6919 http://www.tarbox.org

4 12

Question about slicing
by jorgesmbox-ml＠yahoo.es 16 May '09

16 May '09

Hi, I am just starting with numpy, pyhton and related others. I work on image processing basically. Now, my question is: what is the expected behaviour when slicing a view of an array? The following example might give some background on what I tried to do and the results obatined (which I don't understand): I read an image with PIL but, for whatever reason (different conventions I suppose), it comes upside down. This doesn't change when (I don't know the exact term for this) transforming the image to ndarray with 'array(img)'. I don't feel comfortable working with upside down images, so this had to be fixed. I tried to be smart and avoid copying the whole image: aimg = array(img)[::-1] and it worked!, but I am interested actually in sub-regions of this image, so the next I did was: roi = aimg[10:20,45:50,:] And to my surprise the result was like if I was slicing the original, upside down, image instead of aimg. Can someone explain me what's going on here? I searched and looked at the documentation but I couldn't find an answer. Maybe I am not looking properly. Is the only way to turn the image to perform a copy? Thanks, Jorge

5 6

Leopard install
by Stuart Edwards 16 May '09

16 May '09

Hi I am trying to install numpy 1.3.0 on Leopard 10.5.6 and at the point in the install process where I select a destination, my boot disc is excluded with the message: " You cannot install numpy 1.3.0 on this volume. numpy requires System Python 2.5 to install." I'm not sure what 'System Python 2.5' is as compared to 'Python 2.5' but in the terminal when I type 'python' I get: "Python 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin" so the Python 2.5 requirement seems to be met. Any ideas on what is happening here and why the installer can't see my python installation? (I notice that this issue was raised 3/28 also, but no resolution yet) Thanks for any assistance Stu

5 4

masked ufuncs in C: on github
by Eric Firing 16 May '09

16 May '09

http://www.mail-archive.com/numpy-discussion@scipy.org/msg17595.html Prompted by the thread above, I decided to see what it would take to implement ufuncs with masking in C. I described the result here: http://www.mail-archive.com/numpy-discussion@scipy.org/msg17698.html Now I am starting a new thread. The present state of the work is now in github: http://github.com/efiring/numpy-work/tree/cfastma I don't want to do any more until I have gotten some feedback from core developers. (And I would be delighted if someone wants to help with this, or take it over.) 1) The strategy I have started with is to make a full set of masked ufuncs alongside the existing ones, appending "_m" to their names. Only the binary ufuncs are implemented now, but the unary ufuncs can be handled similarly. Example: multiply(x, y, out) # present ufunc: no change multiply_m(x, y, mask, out) # new Where mask is True, the operation is skipped. 2) I have in mind the possibility of supporting two input masks and one output mask for binary operations. This would look like: multiply_mm(x, y, maskx, masky, out, outmask) outmask would be the logical_or of maskx and masky, and in the case of domained operations it would also be True where the arguments are outside the domain. This form would provide the fastest support for masked arrays, but would also take quite a bit more work, and would expand the namespace even more. I'm not sure it's worth it. 3) I have not yet taken any steps to modify numpy.ma to take advantage of the new ufuncs, but I think that will be quite simple. 4) Likewise, to save time, I am now just borrowing the regular ufunc docstrings. 5) No tests yet, Stefan. They can be added as soon as there is agreement on API and general strategy. 6) The present implementation is based on conceptually small modifications of the existing numpy code generation system. It required a lot of cut and paste, and yields a lot of nearly duplicated code. There may be better ways to do it--especially if it turns out it needs to be redone in some modified form. Eric

5 6

minimal numpy ?
by Robert 16 May '09

16 May '09

for use in binary distribution where I need only basics and fast startup/low memory footprint, I try to isolate the minimal ndarray type and what I need.. with "import numpy" or "import numpy.core.multiarray" almost the whole numpy package tree is imported, _dotblas etc. cxFreeze produces some 10MB numpy baggage (4MB zipped) yet when copying and using the multiarray DLL only, I can create arrays, but he most things fail: >>> import multiarray >>> x=multiarray.array([5,6]) >>> x+x Traceback (most recent call last): File "<interactive input>", line 1, in <module> TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'numpy.ndarray' while this works: >>> b=numpy.core.multiarray.array([3,9]) >>> b+b array([ 6, 18]) I added some things from core.__init__.py like this: import umath import _internal # for freeze programs import numerictypes as nt multiarray.set_typeDict(nt.sctypeDict) .. but the problem of failed type self-recognition remains. What is this? What to do?

2 2

Are masked arrays slower for processing than ndarrays?
by Eli Bressert 15 May '09

15 May '09

Hi, I'm using masked arrays to compute large-scale standard deviation, multiplication, gaussian, and weighted averages. At first I thought using the masked arrays would be a great way to sidestep looping (which it is), but it's still slower than expected. Here's a snippet of the code that I'm using it for. # Computing nearest neighbor distances. # Output will be about 270,000 rows long for the index # and 270,000x50 for the dist array. tree = ann.kd_tree(np.column_stack([l,b])) index, dist = tree.search(np.column_stack([l,b]),k=nth) # Clipping bad values by replacing them acceptable values av[np.where(av<=-10)] = -10 av[np.where(av>=50)] = 50 # Distance clipping and creating mask dist_arcsec = np.sqrt(dist)*3600 mask = dist_arcsec <= d_thresh # Creating masked array av_good = ma.array(av[index],mask=mask) dist_good = ma.array(dist_arcsec,mask=mask) # Reason why I'm using masked arrays. If these were # ndarrays with nan's, then the output would be nan. Std = np.array(np.std(av_good,axis=1)) Var = Std*Std Rho = np.zeros( (len(av), nth) ) Rho2 = np.zeros( (len(av), nth) ) dist_std = np.std(dist_good,axis=1) for j in range(nth): Rho[:,j] = dist_std Rho2[:,j] = Var # This part takes about 20 seconds to compute for a 270,000x50 masked array. # Using ndarrays of the same size takes about 2 second spatial_weight = 1.0 / (Rho*np.sqrt(2*np.pi)) * np.exp( - dist_good / (2*Rho**2)) # Like the spatial_weight section, this takes about 20 seconds W = spatial_weight / Rho2 # Takes less than one second. Ave = np.average(av_good,axis=1,weights=W) Any ideas on why it would take such a long time for processing? Especially the spatial_weight and W variables? Would there be a faster way to do this? Or is there a way that numpy.std can process ignore nan's when processing? Thanks, Eli Bressert

7 14

Re: [Numpy-discussion] (no subject)
by Sebastian Walter 14 May '09

14 May '09

On Wed, May 13, 2009 at 10:18 PM, David J Strozzi <strozzi2(a)llnl.gov> wrote: > Hi, > > [You may want to edit the numpy homepage numpy.scipy.org to tell > people they must subscribe to post, and adding a link to > http://www.scipy.org/Mailing_Lists] > > > Many of you probably know of the interpreter yorick by Dave Munro. As > a Livermoron, I use it all the time. Never heard of it... what does it do? By the sound of it, yorick is an interpreted language like Python. > There are some built-in > functions there, analogous to but above and beyond numpy's sum() and > diff(), which are quite useful for common operations on gridded data. > Of course one can write their own, but maybe they should be cleanly > canonized? > > For instance: > > x = linspace(0,10,10) > y = sin(x) > > It is common, say when integrating y(x), to take "point-centered" > data and want to zone-center it: > > I = sum(zcen(y)*diff(x)) > > def zcen(x): return 0.5*(x[0:-1]+x[1:]) > > Besides zcen, yorick has builtins for "point centering", "un-zone > centering," etc. Also, due to its slick syntax you can give these > things as array "indexes": > > x(zcen), y(dif), z(:,sum,:) > > > Just some thoughts, > David Strozzi > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

3 2

Non-linear optimization in python
by Emerald Jasper 14 May '09

14 May '09

Dear python user! Please, instruct me how to make non-linear optimization using numpy/simpy in python? Thank you very much in the advance, Emerald from Japan

4 3