Mailman 3 February 2006 - NumPy-Discussion

Re: [SciPy-user] Messing with missing values
by Sasha 28 Feb '06

28 Feb '06

Please reply to the list rather than in private e-mail. Private e-mail is likely to end up in a spam folder. See more below. On 2/27/06, pierregm <pgmdevlist(a)mailcan.com> wrote: > Sasha, > Thanks for your answer. > > Ma development page is at > > http://projects.scipy.org/scipy/numpy/wiki/MaskedArray . > > Feel free to add contents there. I would welcome a section listing > > numpy functions that are still not available in ma. > > OK, I'll work on that > Great! > > Median is one of those examples where Paul's recommendation does not > > work because missing values should be ignored rather than filled. For > > example, in R median has two modes: to ignore missing values and to > > return missing value if any value is missing: > > OK, good idea. I already implemented a mamedian for 1d and 2d array. Could you > tell me where I could upload it to be double-checked/tested ? I see three logical locations for patches: 1. Attach to the wiki page: <http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray?action=new>. 2. Post to the list: <mailto:numpy-discussion@lists.sourceforge.net>. 3. Upload to sourceforge: <http://sourceforge.net/tracker/?group_id=1369&atid=301369>. I don't have any preference, but if you choose 1 or 3, please announce the URL on the list. Also it is best to post patches as an output tof "svn diff".

2 1

Re: [Numpy-discussion] missing array type
by Sasha 28 Feb '06

28 Feb '06

On 2/28/06, Colin J. Williams <cjw(a)sympatico.ca> wrote: > ... > >>Would it not be better to have def zeroes(..., zeroStrides= False): > >>and a= zeros(..., zeroStrides= True)? > >> > >> > > > >This is equivalent to what I proposed: xzeros(shape) and xones(shape) > >functions as a shorthand to ndarray(shape, strides=(0,)*len(shape)) > > > > > Wot, more names to remember? [sorry, I can't give you the graphic to go > aling with this. :-) ] Oh, please - you don't have to be so emphatic. Your solution above (adding zeroStrides parameter) would require a much more arbitrary name to remember. Even worse, someone will write zeros(shape, int, False, True) to mean zeros(shape, dtype=int, fortran=False, zeroStrides=True) and anyone reading that code will have to look up the manual to understand what each boolean means. Boolean parameters are generally considered bad design. For example, it would be much better to have array(..., memory_layout='Fortran') instead of current array(...,fortran=True). (Well, fortran=True is not that bad, but fortran=False is really puzzling - if it is not fortran - what is it?) Arguably even better solution would be array(..., strides = fortran_strides(shape)), but that's a different story. The names xzeros and xones are a natural choice because the functionality they provide is very similar to what xrange provides compared to range: memory saving way to achieve the same result.

3 2

can't resize ndarray subclass
by Zachary Pincus 28 Feb '06

28 Feb '06

Thus far, it seems like the way to get instances of ndarray subclasses is to use the 'view(subclass)' method on a proper ndarray, either in the subclass __new__, or after constructing the array. However, subclass instances so created *do not own their own data*, as far as I can tell. They are just "views" on an other array's data. This means that these objects can't be resized, or have other operations performed on them which requires 'owning' the data buffer. E.g.: class f(numpy.ndarray): def __new__(cls, *p, **kw): return numpy.array(*p, **kw).view(cls) f([1,2,3]).resize((10,)) ValueError: cannot resize this array: it does not own its data numpy.array([1,2,3]).view(f).resize((10,)) ValueError: cannot resize this array: it does not own its data numpy.array([1,2,3]).resize((10,)) (no problem) Is there another way to create ndarray subclasses which do own their own data? Note that numpy.resize(f([1,2,3]), (10,)) works fine. But this isn't the same as having an object that owns its data. Specifically, there's just no way to have an ndarray subclass resize itself as a result of calling a method if that object doesn't own its data. (Imagine a variable-resolution polygon type that could interpolate or decimate vertices as needed: such a class would need to resize itself.) Zach

3 8

missing array type
by Alan G Isaac 28 Feb '06

28 Feb '06

The recent discussion of Matlab's repmat plus some recent use of grids leads me to ask: should numpy contain a true repeated array object, where one single copy of the data supports a full set of array operations? (So that, e.g., repmat(x,(3,2)) would simply point to the x data or, if desired, make a single copy of it.) And actually this seems a special case of a truly space-saving Kronecker product. Cheers, Alan Isaac

6 12

Re: Method to shift elements in an array?
by Alan G Isaac 28 Feb '06

28 Feb '06

Tim wrote: > import numpy > def roll(A, n): > "Roll the array A in place. Positive n -> roll right, negative n -> > roll left" > if n > 0: > n = abs(n) > temp = A[-n:] > A[n:] = A[:-n] > A[:n] = temp > elif n < 0: > n = abs(n) > temp = A[:n] > A[:-n] = A[n:] > A[-n:] = temp > else: > pass This probably counts as a gotcha: >>> a=N.arange(10) >>> temp=a[-6:] >>> a[6:]=a[:-6] >>> a[:6]=temp >>> a array([4, 5, 0, 1, 2, 3, 0, 1, 2, 3]) Cheers, Alan Isaac PS Here's something close to the rotater functionality. #rotater: rotate row elements # Format: y = rotater(x,r,copydata) # Input: x RxC array # rotateby size R integer array, or integer (rotation amounts) # inplace boolean (default is False -> copies data) # Output: y RxC array: # rows rotated by rotateby # or None (if inplace=True) # Remarks: Intended for use with 2D arrays. # rotateby values are positive for rightward rotation, # negative for leftward rotation # :author: Alan G Isaac (aisaac AT american DOT edu) # :date: 24 Feb 2006 def rotater(x,rotateby,inplace=False) : assert(len(x.shape)==2), "For 2-d arrays only." xrotate = numpy.array(x,copy=(not inplace)) xrows = xrotate.shape[0] #make an iterater of row shifts if isinstance(rotateby,int): from itertools import repeat rowshifts = repeat(rotateby,xrows) else: rowshifts = numpy.asarray(rotateby) assert(rowshifts.size==xrows) rowshifts = rowshifts.flat #perform rotation on each row for row in xrange(xrows): rs=rowshifts.next() #do nothing if rs==0 if rs>0: xrotate[row] = numpy.concatenate([xrotate[row][-rs:],xrotate[row][:-rs]]) elif rs<0: xrotate[row] = numpy.concatenate([xrotate[row][:-rs],xrotate[row][-rs:]]) if inplace: return None else: return xrotate

2 1

Re: [Numpy-discussion] wiki page for record arrays
by Joe Harrington 28 Feb '06

28 Feb '06

I fixed a small error. I found myself a bit lost. Are cookbook pages supposed to be introductory, or are they aimed at users who already know a fair bit? From the page, I can vaguely get that record arrays allow you access to your data using text strings as partial indexers, but I found myself focusing so much on testing whether this was true and figuring out how they work that I was completely distracted from the example. A paragraph or two at the top explaining what a record array is, why it's useful, and what the basic properties are would be good. Then give the example. Also, it's less than intuitive why you are storing RGB values (usually thought of as integers in the range 0-255 or 0x00 - 0xff) in 32-bit floating-point numbers. --jh--

2 1

Simplest ndarray subclass __new__ possible?
by Zachary Pincus 28 Feb '06

28 Feb '06

Hi folks, I'm trying to write an ndarray subclass with a constructor like the matrix constructor -- one which can take matrix objects, array objects, or things that can be turned into array objects. I've copied the __new__ method from matrix (and tried to eliminate the matrix-specific stuff), but there's a lot of code there. So I'm trying to figure out what the absolute minimum I need is for correct behavior. (This would be a useful wiki entry somewhere. In fact, a whole page about subclassing ndarray would be good.) What follows is what I have so far. Have I missed anything, or can anything else be removed? Zach class contour(numpy.ndarray): def __new__(subtype, data, dtype=None, copy=True): ##### Do I need this first if block? ##### Wouldn't the second block would do fine on its own? if isinstance(data, contour): dtype2 = data.dtype if (dtype is None): dtype = dtype2 if (dtype2 == dtype) and (not copy): return data return data.astype(dtype) if isinstance(data, numpy.ndarray): if dtype is None: intype = data.dtype else: intype = numpy.dtype(dtype) new = data.view(contour) if intype != data.dtype: return new.astype(intype) if copy: return new.copy() else: return new # now convert data to an array arr = numpy.array(data, dtype=dtype, copy=copy) ##### Do I need this if block? if not (arr.flags.fortran or arr.flags.contiguous): arr = arr.copy() ##### Do I need the fortran flag? ret = numpy.ndarray.__new__(subtype, arr.shape, arr.dtype, buffer=arr, fortran=arr.flags.fortran) return ret

3 5

Faster fill
by Sasha 27 Feb '06

27 Feb '06

I''ve just posted a patch at http://projects.scipy.org/scipy/numpy/wiki/PossibleOptimizationAreas that results in a 6-14x speed-up of ndarray.fill for simple datatypes. That's a bigger change than what I am confortable submiting to svn without a review. Also since I am not familiar with record arrays, I am not sure whether or not this change would break anything in that area. Finally, the patch only adresses a single-segment case, "strided" arrays would still use old code.

3 3

[SciPy-user] Messing with missing values
by Sasha 27 Feb '06

27 Feb '06

I am replying on "numpy-discussion" because this is really a numpy rather than scipy topic. > Unfortunately, most of the numpy/scipy functions don't handle missing values > nicely. Can you specify which *numpy* functions are giving you trouble? That should be fixed. > How could I mask the values corresponding to > MA.masked in the final list, without having to check every single element? Latest ma allows you to pass masked arrays directly to ufuncs. In order for this to work a ufunc should be registered in the "domains" and "fills" dictionaries. Not much documentation on this feature exists yet, so you will have to read the code in ma.py to figure this out. > Date: Sat, 25 Feb 2006 18:36:19 -0500 > From: pgmdevlist(a)mailcan.com > Subject: [SciPy-user] Messing with missing values > To: scipy-user(a)scipy.net > Message-ID: <200602251836.20406.pgmdevlist(a)mailcan.com> > Content-Type: text/plain; charset="us-ascii" > > Folks, > Most of the data I work with have missing values, and I rely on 'ma' a lot. > Unfortunately, most of the numpy/scipy functions don't handle missing values > nicely. Not a problem I thought, I just have to adapt the functions I need. > I have 2 options: wrapping the initial functions in tests, or recode the > initial functions. > * According to your experience, which is the most efficient way to go ? > * I have a function that outputs either a float or MA.masked. I call this > function recursively, appending the result to a list, and then trying to > process the list as an array. How could I mask the values corresponding to > MA.masked in the final list, without having to check every single element? > > Thanks for your ideas

3 3

bug report: nonzero on masked arrays
by Andrew Straw 27 Feb '06

27 Feb '06

I know there's been some discussion along these lines lately, but I haven't followed it closely and thought I could still give a simple bug report. I can post this on Trac if no one can tackle it immediately -- let me know. --Andrew In [1]: import numpy In [2]: numpy.__file__ Out[2]: '/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2172-py2.3-linux-x86_64.egg/numpy/__init__.pyc' In [3]: jj= numpy.ma.masked_array( [0,1,2,3,0,4,5,6], mask=[0,0,0,1,1,1,0,0]) In [4]: numpy.ma.nonzero(jj) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /home/astraw/src/kookaburra/flydra/analysis/<ipython console> /home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2172-py2.3-linux-x86_64.egg/numpy/core/ma.py in __call__(self, a, *args, **kwargs) 317 else: 318 if m.shape != shape: --> 319 m = mask_or(getmaskarray(a), getmaskarray(b)) 320 return masked_array(result, m) 321 NameError: global name 'b' is not defined In [5]:

1 0