Eric Firing wrote:

Sasha wrote:

...

In an ideal world, any function that accepts ndarray would accept ma.array and vice versa. Moreover, if the ma.array has no masked elements and the same data as ndarray, the result should be the same.

This would be *very* nice.

...

Obviously current implementation falls short of this goal, but there is one feature that seems to make this goal unachievable.

This feature is the "filled" method of ma.array. Pydoc for this method reports the following:

| filled(self, fill_value=None) | A numeric array with masked values filled. If fill_value is None, | use self.fill_value(). | | If mask is nomask, copy data only if not contiguous. | Result is always a contiguous, numeric array. | # Is contiguous really necessary now?

That is not the best possible description ("filled" is "filled"), but the essence is that the result of a.filled(value) is a contiguous ndarray obtained from the masked array by copying non-masked elements and using value for masked values.

I would like to propose to add a "filled" method to ndarray. I see several possibilities and would like to hear your opinion:

1. Make filled simply return self.

2. Make filled return a contiguous copy.

3. Make filled replace nans with the fill_value if array is of floating point type.

It seems to me that any function or method that returns an array from an array should be perfectly consistent and explicit about whether it makes a copy or not. Sometimes the filled method *needs* to return a copy; therefore it should *always* return a copy, regardless of the presence or state of masking. Hence I think the filled method of ma needs to be changed in this way also.

+1 (mumble mumble reshape mumble)

The question for your suggestion 3 is, should a nan always be the equivalent of a masked value? One loses a little flexibility, but it has an appealing simplicity to it. I could be persuaded otherwise, but right now I would vote for it.

-0 (I'd be -1, but I don't really use masked arrays). While masked array should be convenient for masking. The basic ndarray object should be as flexible as possible since it's the building block for all the rest. -tim

On 3/22/06, Travis Oliphant wrote:

Tim makes a good point here. Should the reshape method be fixed to always return a copy? The semantics a.shape = (...) could still be used to re-shape contiguous arrays where possible.

Reshape never copies the data:

...

...

x = ones(4) x.reshape((2,2)).__array_data__ == x.__array_data__ True

The only inconsistency is that

...

...

x.reshape((2,2)) is x False x.reshape((4,)) is x True

I agree that this is unnecessary, but don't see much of a problem. +0 here

However, whether or not reshape returns a copy is consistent (but perhaps not explicitly explained).

To me consistency means "is independent of the input." Whether or not reshape creates a new python object depends on the value of the argument. I would call it inconsistency.

We will still have .ravel() which sometimes copies and sometimes doesn't.

Ravel should be a shortcut for x.reshape((x.size,)), so it is really the same question. +0 (to make ravel always return a new python object)

Tim Hochberg wrote:

My opinion is that all methods and functions should either:

1. Always return a copy. 2. Always return a view 3. Return a view if possible otherwise raise an exception.

Well, but is copy/view the /only/ invariant worth guaranteeing? I think there is a valid need for functions which ensure other invariants, such as contiguity. There are applications (such as passing pointers to C/Fortran libraries which don't have striding mechanisms but will not modify their inputs) which require contiguous inputs, but where one would rather make a copy only if necessary. My take on this is that we should /document/ clearly what invariants any given function satisfies, but I think the 'always view/always copy' view excludes an important usage case. There may be others beyond contiguity, but that's the one that pops immediately to mind. Cheers, f

Tim Hochberg wrote:

I don't think we're in disagreement here although I'm not sure.

I will add, on the subject of continuity, that I think there should be a function 'ascontiguous' that parallels asarray, but assures that the result is contiguous. Although this sometimes returns a copy, I thinks that's OK since that's it's job. I would like to see all of the implicit copying pushed into functions like asarray and ascontiguous.

I think we agree: something like ascontiguous() is precisely what I had in mind (I think that's what ravel() does today, but I'm fine if it gets a new name, as long as the functionality exists). Obviously a function like this should explicitly (docstring) say that it does NOT make any guarantees about whether its return value is a view or a copy, just that it's contiguous. Cheers, f

Tim Hochberg wrote:

a.ravel() seems to be equivalent to reshape(a, [-1]). That is, it returns a flattened, contiguous copy. ascontiguous(a) would be slightly different in that it would preserves the shape of a. In fact I think it would look a lot like:

def ascontiguous(a): """ascontiguous(a) -> contiguous representation of a.

If 'a' is allready contiguous, it is returned unchanged. Otherwise, a contiguous copy is returned.

""" a = asarray(a) if not a.flags['CONTIGUOUS']: a = array(a) return a

Yup, sorry: I've been thinking too much in terms of raw C (double*) pointers so to me, these days everything is 1-d :) I forgot about shape issues... But I think we agree on the basic point. Cheers, f

Tim Hochberg wrote:

I was just looking at the interface for array and asarray to see what other stuff should go in the interface of the hypothetical ascontiguous. There's 'dtype', which I knew about, and 'fortran', which I didn't, but which makes sense. However, there's also 'ndmin'. First off, it's not described in docstring for asarray, but I was able to find it in the docstring for array without a problem. Second, is it really necessary? It seems to be useful in an awfully narrow set of circumstances, particularly since when you are padding axes not everyone wants to pad to the left.

Padding to the left is "default" behavior for broadcasting and so it seems appropriate. This is how all lower-dimensional arrays are interpreted as "higher" dimensional arrays throughout the code. The ndmin is very handy as atested to by the uses of atleast_1d or atleast_2d in numpy library code. It was added later as an optimization step because of the number of library routines that were using it. I've since used it several times to simplify code. I think an ascontinguous on the Python level is appropriate since such a beast exists on the C-level. On the other hand, while Tim prefers narrow interfaces, the array_from_object interface is traditionally pretty broad. Thus, in my mind, the array call should get another flag keyword that forces a contiguous result. This translates easily to the C-domain, in much the same way as the fortran keyword does. -Travis

Tim Hochberg wrote:

That makes some sense.

...

The ndmin is very handy as atested to by the uses of atleast_1d or atleast_2d in numpy library code. It was added later as an optimization step because of the number of library routines that were using it. I've since used it several times to simplify code.

OK, I'll take your word for it.

Not necessarily sane advice :-) --- I might be overstating things. I know that atleast_1d and atleast_2d are used all over the place in scipy. This makes sense and I can certainly understand it. I'm willing to modify things to give narrow interfaces. Right now, since the requesting both fortran and contiguous does not make sense, setting the fortran flag to false enforces C-style contiguous while setting it to True enforces fortran-style. Setting it to None (the default) specifies you don't care and the behavior will be to create C-style contiguous for new arrays and use the striding specified by the array if it's already an array. I admit that it is arguable whether or not the fortran flag should be overloaded like this or not. There are now ascontiguous and asfortran functions with fairly minimal interfaces to make it simpler. There is also a check to make sure that array is called with no more than 2 non keyword arguments. Thus, you won't be able to confuse which flag is which. -Travis

Sasha wrote:

On 3/23/06, Travis Oliphant wrote:

...

Thus, in my mind, the array call should get another flag keyword that forces a contiguous result.

Please don't! The fortran flag is bad enough, but has too much history behind it. Let's not breed boolean parameters. Sooner or later someone will use keword arguments positionally and you will end up guessing what

There are several boolean flags in the interface already. Adding another one won't change the current situation that you describe. There are several ways to handle this. For one, we could force the use of keyword arguments, so that the position problem does not arise. Sasha has mentioned in the past a strides array argument, but I think the default fortran and contiguous strides cases need better support then just one of many possible stridings so I wouldn't go that direction here. I'm debating whether or not the fortran flag should be used to specify both contiguous and fortran cases. Right now, the fortran argument is a three-case flag with dont-care, True, and False arguments. It seems natural to have True mean force-fortran and False mean force-contiguous with dont-care (the default) mean take an array already given (or create a C-contiguous array if we are generating a new array from another object). At any rate, if the fortran flag is there, we need to specify the contiguous case as well. So, either propose a better interface (we could change it still --- the fortran flag doesn't have that much history) to handle the situation or accept what I do ;-) -Travis

Sasha wrote:

On 3/23/06, Travis Oliphant wrote:

...

At any rate, if the fortran flag is there, we need to specify the contiguous case as well. So, either propose a better interface (we could change it still --- the fortran flag doesn't have that much history) to handle the situation or accept what I do ;-)

Let me try. I propose to eliminate the fortran flag in favor of a more general "strides" argument. This argument can be either a sequence of integers that becomes the strides, or a callable object that takes shape and dtype arguments and return a sequence that becomes the strides. For fortran and c order functions that generate appropriate stride sequences should be predefined to enable array(..., strides=fortran, ...) and array(..., strides=contiguous).

I like the idea of being able to create an array with custom strides. The applications aren't entirely clear yet, but it does seem like it could have some interesting and useful consequences. That said, I don't think this belongs in 'array'. Historically, array has been used for all sorts of array creation activities, which is why it always seems to have a wide, somewhat incoherent interface. However, most uses of array() boil down to one thing: creating a *new* array from a python object. My preference would be to focus on that functionality for array() and spin of it's other historical uses and new uses, like this custom strided array stuff, into separate factory functions. For example (and just for example, I make no great claims for either this name or interface): a = array_from_data(a_buffer_object, dtype, dims, strides) One thing that you do make clear is that contiguous and fortran should really two values of the same flag. If you combine this with one other simplification: array() always copies, we end up with a nice thin interface: # Create a new array in 'order' order. Defaults to "C" order. array(object, dtype=None, order="C"|"FORTRAN") and # Returns an array. If object is an array and order is satisfied, return object otherwise a new array. # If order is set the returned array will be contiguous and have that ordering asarray(object, dtype=None, order=None|"C"|"FORTRAN") # Just the same, but allow subtypes. asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") You could build asarray, asanyarray, etc on top of the proposed array without problems by using type(object)==ndarray and isinstance(type, ndarray) respectively. Stuff like convenience functions for minnd would also be easy to build on top of there. This looks great to me (pre-coffee). Embrace simplicity: you have nothing to lose but your clutter;) Regards, -tim

Colin J. Williams wrote:

Tim Hochberg wrote:

...

Sasha wrote:

...

On 3/23/06, Travis Oliphant wrote:

...

At any rate, if the fortran flag is there, we need to specify the contiguous case as well. So, either propose a better interface (we could change it still --- the fortran flag doesn't have that much history) to handle the situation or accept what I do ;-)

Contiguity is separable from fortran: [Dbg]>>> b= _n.array([[1, 2, 3], [4, 5, 6]]) [Dbg]>>> b.flags.contiguous True [Dbg]>>> c= b.transpose() [Dbg]>>> c array([[1, 4], [2, 5], [3, 6]]) [Dbg]>>> c.flags.contiguous False [Dbg]>>>

This is true, but irrelevant. To the best of my knowledge, the only reason to force an array to be in a specific order is to pass it to a C function that expects either FORTRAN- or C-ordered arrays. And, in that case, the array also needs to be contiguous. So, for the purpose of creating arrays (and for the purposes of ascontiguous), the only cases that matter are arrays that are both contiguous and the specified order. Thus, specifying continuity and order separately to the constructor needlessly complicates the interface. Or since I'm feeling jargon happy today, YAGNI.

...

...

Let me try. I propose to eliminate the fortran flag in favor of a more general "strides" argument. This argument can be either a sequence of integers that becomes the strides, or a callable object that takes shape and dtype arguments and return a sequence that becomes the strides. For fortran and c order functions that generate appropriate stride sequences should be predefined to enable array(..., strides=fortran, ...) and array(..., strides=contiguous).

I like the idea of being able to create an array with custom strides. The applications aren't entirely clear yet, but it does seem like it could have some interesting and useful consequences. That said, I don't think this belongs in 'array'. Historically, array has been used for all sorts of array creation activities, which is why it always seems to have a wide, somewhat incoherent interface. However, most uses of array() boil down to one thing: creating a *new* array from a python object. My preference would be to focus on that functionality for array() and spin of it's other historical uses and new uses, like this custom strided array stuff, into separate factory functions. For example (and just for example, I make no great claims for either this name or interface): a = array_from_data(a_buffer_object, dtype, dims, strides) [***]

One thing that you do make clear is that contiguous and fortran should really two values of the same flag.

Please see the transpose example above.

...

If you combine this with one other simplification: array() always copies, we end up with a nice thin interface: # Create a new array in 'order' order. Defaults to "C" order. array(object, dtype=None, order="C"|"FORTRAN")

I feel that [***] above is much cleaner than this. I suggest that string constants be deprecated.

I'm no huge fan of string constants myself, but I think you need to think this through more. First off, the interface I tossed off above doesn't cover the same ground as array, since it works off an already created buffer object. That means you'd have to go through all sorts of contortions and do at least one copy to get data into Fortran order. You could allow arbitrary, 1D, python sequences instead, but that doesn't help the common case of converting a 2D python object into a 2D array. You could allow N-D python objects, but then you have two ways of specifying the dims of the object and things become a big krufty mess. Compared to that string constants are great.

...

and # Returns an array. If object is an array and order is satisfied, return object otherwise a new array. # If order is set the returned array will be contiguous and have that ordering asarray(object, dtype=None, order=None|"C"|"FORTRAN") # Just the same, but allow subtypes. asanyarray(object, dtype=None, order=None|"C"|"FORTRAN")

You could build asarray, asanyarray, etc on top of the proposed array without problems by using type(object)==ndarray and isinstance(type, ndarray) respectively. Stuff like convenience functions for minnd would also be easy to build on top of there. This looks great to me (pre-coffee).

Embrace simplicity: you have nothing to lose but your clutter;)

Regards,

-tim

If [***] above were adopted, it would still be helpful to adopt numarray's iscontiguous method, or better, use a property.

-0. In my experience, 99% of my use cases would be covered for ascontiguous and for the remaining 1% I'm happy to use a.flags.contiguous. Regards, -tim

Travis Oliphant wrote:

Tim Hochberg wrote:

...

...

Please see the transpose example above.

...

If you combine this with one other simplification: array() always copies, we end up with a nice thin interface: # Create a new array in 'order' order. Defaults to "C" order. array(object, dtype=None, order="C"|"FORTRAN")

I like the order flag. It's a relatively easy switch and one we should make right now. I don't mind string constants either in this case.

Great!

Removing the copy flag will break a lot of code because it's been around for a long time. This is also not an "easy thing" to add to convertcode.py though I suppose 90% of the cases could be found.

I kinda figured on that. But I figured I'd propose my favorite and see what came of it.

We would also have to re-write asarray to be an additional C-function to make it not copy but make array copy.

I thought so too at first, but I don't this is is so. Untested, and can could probably be cleaned up some: def asarray(obj, order=None): if type(obj) == ndarray: if order: if not obj.flags.contiguous: return array(obj, order) if order == "C" and obj.flags.fortran: return array(obj, order) if order == "FORTRAN" and not obj.flags.fortran: return array(obj, order) return obj else: if order: return array(obj, order) else: return array(obj) For asanyarray, simply replace the type test with an isinstance test.

So, for now I'm not as enthused about that idea.

Yeah. Without backward compatibility constraints I'm convinced that it's the right thing to do, but I realize there is a need to balance making the transistion manageable with making things "perfect". -tim

On 3/24/06, Tim Hochberg wrote:

.... For example (and just for example, I make no great claims for either this name or interface): a = array_from_data(a_buffer_object, dtype, dims, strides)

This looks very similar to the current ndarray "new" constructor: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) | | There are two modes of creating an array using __new__: | 1) If buffer is None, then only shape, dtype, and fortran | are used | 2) If buffer is an object exporting the buffer interface, then | all keywords are interpreted. | The dtype parameter can be any object that can be interpreted | as a numpy.dtype object. (see pydoc numpy.ndarray) I would not mind to leave array() unchanged and move discussion to streamlining ndarray.__new__ . For example, some time ago I suggested that strides should be interpreted even if buffer=None.

Alexander Belopolsky wrote:

On 3/24/06, Tim Hochberg wrote:

...

.... For

example (and just for

...

example, I make no great claims for either this name or interface): a = array_from_data(a_buffer_object, dtype, dims, strides)

This looks very similar to the current ndarray "new" constructor:

It sure does.

| ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) | | There are two modes of creating an array using __new__: | 1) If buffer is None, then only shape, dtype, and fortran | are used | 2) If buffer is an object exporting the buffer interface, then | all keywords are interpreted. | The dtype parameter can be any object that can be interpreted | as a numpy.dtype object.

(see pydoc numpy.ndarray)

I would not mind to leave array() unchanged and move discussion to streamlining ndarray.__new__ . For example, some time ago I suggested that strides should be interpreted even if buffer=None.

That does look like a good place to hang any arbitrary strides creating stuff. I'm of the opinion that arguments should *never* be ignored, so I'm all for interpreting strides even when buffer is None. I'd also contend that offset should either be respected (by overallocating) or since that's probably useless, raising a ValueError when it's nonzero. Regards, Tim

Sasha wrote:

On 3/23/06, Travis Oliphant wrote:

...

Thus, in my mind, the array call should get another flag keyword that forces a contiguous result.

Please don't! The fortran flag is bad enough, but has too much history behind it. Let's not breed boolean parameters. Sooner or later someone will use keword arguments positionally and you will end up guessing what

array([1,2], int8_, 1, 1, 0, 0)

means.

We could always force keyword parameter by using **kwd. That would mess mess up help() and other introspection tools. Then again, if we want to discourage overuse of this sort of stuff, perhaps that's not a bad thing ;) But if we really need access to the C guts of an array, just allow a dictionary of flags to get plugged in. These would be the same as what somearray.flags puts out. The new interface for array would be: array(object, **flags) Where flags could contain: CONTIGUOUS : force a copy if object isn't contiguous if True. [default None] FORTRAN : force array to be fortran order if True, C order if False [default None] OWNDATA : force a copy if True [default True] WRITEABLE : Force a copy if object isn't writeable if Tru [default None] ALIGNED : Force a copy if object isn't aligned if True. [default None] UPDATEIFCOPY : Set the UPDATEIFCOPY flag? [default ???] With the exception of the FORTRAN, and possibly UPDATEIFCOPY, it would be an error to set any of these flags to False (Forcing an array to be discontiguous for instance, makes no sense). That's a thin interface and it ties together with the flags parameter nicely. On the downside, it's a little weird, particularly using OWNDATA for copy, although it is logical once you think about it. It also drops 'minnd' and 'subok'. I wouldn't miss them, but I expect someone would squawk. You could shoehorn them into flags, but then you lose one of the chief virtues of this scheme, which is that it makes a strong connection between the constructor and the flags parameter. 'subok' should be pretty well taken care of by 'asanyarray' and it would be easy enough to create an auxilliary function to replicate the 'minnd' functionality. With this kind of narrow interface, it might make sense to allow the flags parameter on all of the asX functions. That makes for a nice uniform, easy to remember, interface. I'm not positive I like this idea yet, but I thought it was interesting enough to throw into the ring anyway. Tangentially, I noticed that I can set ALIGNED to False. Isn't that going to break something somewhere? Should the ALIGNED flag be readonly? -tim

Sasha wrote:

On 3/23/06, Travis Oliphant wrote:

...

Thus, in my mind, the array call should get another flag keyword that forces a contiguous result.

Please don't!

+1

The fortran flag is bad enough, but has too much history behind it. Let's not breed boolean parameters. Sooner or later someone will use keword arguments positionally and you will end up guessing what

array([1,2], int8_, 1, 1, 0, 0)

means.

The use of keyword parameters can help here. Various editors use them to provide tips and there is the help function. Colin W.

There is the flatten() method which exists for precisely this reason (it *always* returns a copy). -Travis

On 4/7/06, Tim Hochberg wrote:

... In general, I'm skeptical of adding more methods to the ndarray object -- there are plenty already.

I've also proposed to drop "fill" in favor of optimizing x[...] = <scalar>. Having both "fill" and "filled" in the interface is plain awkward. You may like the combined proposal better because it does not change the total number of methods :-) In addition, it appears that both the method and function versions of

filled are "dangerous" in the sense that they sometimes return the array itself and sometimes a copy.

This is true in ma, but may certainly be changed.

Finally, changing ndarray to support masked array feels a bit like the tail wagging the dog.

I disagree. Numpy is pretty much alone among the array languages because it does not have "native" support for missing values. For the floating point types some rudimental support for nans exists, but is not really usable. There is no missing values machanism for integer types. I believe adding "filled" and maybe "mask" to ndarray (not necessarily under these names) could be a meaningful step towards "native" support for missing values.

Eric Firing wrote:

Sasha wrote:

...

On 4/7/06, *Tim Hochberg* mailto:tim.hochberg@cox.net> wrote:

... In general, I'm skeptical of adding more methods to the ndarray object -- there are plenty already.

I've also proposed to drop "fill" in favor of optimizing x[...] = <scalar>. Having both "fill" and "filled" in the interface is plain awkward. You may like the combined proposal better because it does not change the total number of methods :-)

In addition, it appears that both the method and function versions of filled are "dangerous" in the sense that they sometimes return the array itself and sometimes a copy.

This is true in ma, but may certainly be changed.

Finally, changing ndarray to support masked array feels a bit like the tail wagging the dog.

I disagree. Numpy is pretty much alone among the array languages because it does not have "native" support for missing values. For the floating point types some rudimental support for nans exists, but is not really usable. There is no missing values machanism for integer types. I believe adding "filled" and maybe "mask" to ndarray (not necessarily under these names) could be a meaningful step towards "native" support for missing values.

I agree strongly with you, Sasha. I get the impression that the world of numerical computation is divided into those who work with idealized "data", where nothing is missing, and those who work with real observations, where there is always something missing.

I think your experience is clouding your judgement here. Or at least this comes off as unnecessarily perjorative. There's a large class of people who work with data that doesn't have missing values either because of the nature of data acquisition or because they're doing simulations. I take zillions of measurements with digital oscillopscopes and they *never* have missing values. Clipped values, yes, but even if I somehow could queery the scope about which values were actually clipped or simply make an educated guess based on their value, the facilities of ma would be useless to me. The clipped values are what I would want in any case. I also do a lot of work with simulations derived from this and other data. I don't come across missing values here but again, if I did, the way ma works would not help me. I'd have to treat them either by rejecting the data outright or by some sort of interpolation.

As an oceanographer, I am solidly in the latter category. If good support for missing values is not built in, it has to be bolted on, and it becomes clunky and awkward.

This may be a false dichotomy. It's certainly not obvious to me that this is so. At least if "bolted on" means "not adding a filled method to ndarray".

I was reluctant to speak up about this earlier because I thought it was too much to ask of Travis when he was in the midst of putting numpy on solid ground. But I am delighted that missing value support has a champion among numpy developers, and I agree that now is the time to change it from "bolted on" to "integrated".

I have no objection to ma support improving. In fact I think it would be great although I don't forsee it helping me anytime soon. I also support Sasha's goal of being able to mix MaskedArrays and ndarrays reasonably seemlessly. However, I do think the situation needs more thought. Slapping filled and mask onto ndarray is the path of least resistance, but it's not clear that it's the best one. If we do decide we are going to add both of these methods to ndarray (with filled returning a copy!), then it may worth considering making ndarray a subclass of MaskedArray. Conceptually this makes sense, since at this point an ndarray will just be a MaskedArray where mask is always False. I think that they could share much of the implementation except that ndarray would be set up to use methods that ignored the mask attribute since they would know that it's always false. Even that might not be worth it, since the check for whether mask is True/False is just a pointer compare. It may in fact be best just to do away with MaskedArray entirely, moving the functionality into ndarray. That may have performance implications, although I don't seem them at the moment, and I don't know if there are other methods/attributes that this would imply need to be moved over, although it looks like just mask, filled and possibly filled_value, although the latter looks a little dubious to me. Either of the above two options would certainly improve the quality of MaskedArray. Copy for instance seems not to have been implemented, and who knows what other dark corners remain unexplored here. There's a whole spectrum of possibilities here from ones that don't intrude on ndarray at all to ones that profoundly change it. Sasha's suggestion looks like it's probably the simplest thing in the short term, but I don't know that it's the best long term solution. I think it needs more thought and discussion, which is after all what Sasha asked for ;) Regards, -tim

On 4/7/06, Tim Hochberg wrote:

[...]

However, I do think the situation needs more thought. Slapping filled and mask onto ndarray is the path of least resistance, but it's not clear that it's the best one.

Completely agree. I have many gripes about current ma implementation of both "filled" and "mask". filled: 1. I don't like default fill value. It should be mandatory to supply fill value. 2. It should return masked array (with trivial mask), not ndarray. 3. The name conflicts with the "fill" method. 4. View/Copy inconsistency. Does not provide a method to fill values in-place. mask: 1. I've got rid of mask returning None in favor of False_ (boolean array scalar), but it is still not perfect. I would prefer data.shape == mask.shape invariant and if space saving/performance is deemed necessary use zero-stride arrays. 2. I don't like the name. "Missing" or "na" would be better.

If we do decide we are going to add both of these methods to ndarray (with filled returning a copy!), then it may worth considering making ndarray a subclass of MaskedArray. Conceptually this makes sense, since at this point an ndarray will just be a MaskedArray where mask is always False. I think that they could share much of the implementation except that ndarray would be set up to use methods that ignored the mask attribute since they would know that it's always false. Even that might not be worth it, since the check for whether mask is True/False is just a pointer compare.

The tail becoming the dog! Yet I agree, this makes sense from the implementation point of view. From OOP perspective this would make sense if arrays were immutable, but since mask is settable in MaskedArray, making it constant in the subclass will violate the substitution principle. I would not object making mask read only, however.

It may in fact be best just to do away with MaskedArray entirely, moving the functionality into ndarray. That may have performance implications, although I don't seem them at the moment, and I don't know if there are other methods/attributes that this would imply need to be moved over, although it looks like just mask, filled and possibly filled_value, although the latter looks a little dubious to me.

I think MA can coexist with ndarray and share the interface. Ndarray can use special bit-patterns like IEEE NaN to indicate missing floating point values. Add-on modules can redefine arithmetic to make INT_MIN behave as a missing marker for signed integers (R, K and J (I think) languages use this approach). Applications that need missing values support across the board will use MA.

Either of the above two options would certainly improve the quality of MaskedArray. Copy for instance seems not to have been implemented, and who knows what other dark corners remain unexplored here.

More (corners) than you want to know about! Reimplementing MA in C would be a worthwhile goal (and what you suggest seems to require just that), but it is too big of a project. I suggest that we focus on the interface first. If existing MA interface is rejected (which is likely) for ndarray, we can easily experiment with the alternatives within MA, which is pure python.

There's a whole spectrum of possibilities here from ones that don't intrude on ndarray at all to ones that profoundly change it. Sasha's suggestion looks like it's probably the simplest thing in the short term, but I don't know that it's the best long term solution. I think it needs more thought and discussion, which is after all what Sasha asked for ;)

Exactly!

Sasha wrote:

On 4/7/06, Tim Hochberg wrote:

...

[...]

However, I do think the situation needs more thought. Slapping filled and mask onto ndarray is the path of least resistance, but it's not clear that it's the best one.

Completely agree. I have many gripes about current ma implementation of both "filled" and "mask".

filled:

1. I don't like default fill value. It should be mandatory to supply fill value.

That makes perfect sense. If anything should have a default fill value, it's the functsion calling filled, not the arrays themselves.

2. It should return masked array (with trivial mask), not ndarray.

So, just with mask = False? In a follow on message Pierre disagress and claims that what you really want is the ndarray since not everything will accept. Then I guess you'd need to call b.filled(fill).data. I agree with Sasha in principle but Pierre, perhaps in practice. I'm almost suggested it get renames a.asndarray(fill), except that asXXX has the wrong conotations. I think this one needs to bounce around some more.

3. The name conflicts with the "fill" method.

I thought you wanted to kill that. I'd certainly support that. Can't we just special case __setitem__ for that one case so that the performance is just as good if performance is really the issue?

4. View/Copy inconsistency. Does not provide a method to fill values in-place.

b[b.mask] = fill_value; b.unmask() seems to work for this purpose. Can we just have filled return a copy?

mask:

1. I've got rid of mask returning None in favor of False_ (boolean array scalar), but it is still not perfect. I would prefer data.shape == mask.shape invariant and if space saving/performance is deemed necessary use zero-stride arrays.

Interesting idea. Is that feasible yet?

2. I don't like the name. "Missing" or "na" would be better.

I'm not on board here, although really I'd like to here from other people who use the package. 'na' seems to cryptic to me and 'missing' to specific -- there might be other reasons to mask a value other it being missing. The problem with mask is that it's not clear whether True means the data is useful or unuseful. Keep throwing out names, maybe one will stick.

...

If we do decide we are going to add both of these methods to ndarray (with filled returning a copy!), then it may worth considering making ndarray a subclass of MaskedArray. Conceptually this makes sense, since at this point an ndarray will just be a MaskedArray where mask is always False. I think that they could share much of the implementation except that ndarray would be set up to use methods that ignored the mask attribute since they would know that it's always false. Even that might not be worth it, since the check for whether mask is True/False is just a pointer compare.

The tail becoming the dog! Yet I agree, this makes sense from the implementation point of view. From OOP perspective this would make sense if arrays were immutable, but since mask is settable in MaskedArray, making it constant in the subclass will violate the substitution principle. I would not object making mask read only, however.

How do you set the mask? I keep getting attribute errors when I try it. And unmask would be a noop on an ndarray.

...

It may in fact be best just to do away with MaskedArray entirely, moving the functionality into ndarray. That may have performance implications, although I don't seem them at the moment, and I don't know if there are other methods/attributes that this would imply need to be moved over, although it looks like just mask, filled and possibly filled_value, although the latter looks a little dubious to me.

I think MA can coexist with ndarray and share the interface. Ndarray can use special bit-patterns like IEEE NaN to indicate missing floating point values. Add-on modules can redefine arithmetic to make INT_MIN behave as a missing marker for signed integers (R, K and J (I think) languages use this approach). Applications that need missing values support across the board will use MA.

...

Either of the above two options would certainly improve the quality of MaskedArray. Copy for instance seems not to have been implemented, and who knows what other dark corners remain unexplored here.

More (corners) than you want to know about! Reimplementing MA in C would be a worthwhile goal (and what you suggest seems to require just that), but it is too big of a project. I suggest that we focus on the interface first. If existing MA interface is rejected (which is likely) for ndarray, we can easily experiment with the alternatives within MA, which is pure python.

Perhaps MaskedArray should inherit from ndarray for the time being. Many of the methods would need to reimplemented anyway, but it would make asanyarray work. Someone was just complaining about asarray munging his arrays. That's correct behaviour, but it would be nice if asanyarray did the right thing. I suppose we could just special case asanyarray to ignore MaskedArrays, that might be better since it's less constraining from an implementation side too.

...

There's a whole spectrum of possibilities here from ones that don't intrude on ndarray at all to ones that profoundly change it. Sasha's suggestion looks like it's probably the simplest thing in the short term, but I don't know that it's the best long term solution. I think it needs more thought and discussion, which is after all what Sasha asked for ;)

Exactly!

This may be an oportune time to propose something that's been cooking in the back of my head for a week or so now: A stripped down array superclass. The details of this are not at all locked down, but here's a strawman proposal. We add an array superclass. call it basearray, that has the same C-structure as the existing ndarray. However, it has *no* methods or attributes. It's simply a big blob of data. Functions that work on the C structure of arrays (ufuncs, etc) would still work on this arrays, as would asarray, so it could be converted to an ndarray as necessary. In addition, we would supply a minimal set of functions that would operate on this object. These functions would be chosen so that the current array interface could be implemented on top of them and the basearray object in pure python. These functions would be things like set_shape(a, shape), etc. They would be segregated off in their own namespace, not in the numpy core. [Note that I'm not proposing we actually implement ndarray this way, just that we make is possible]. This leads to several useful outcomes. 1. If we're careful, this could be the basic array object that we propose, at least for the first roun,d for inclusion in the Python core. It's not useful for anything but passing data betwen various application that understand the data structure, but that in itself could be a huge win. And the fact that it's dirt simple would probably be an advantage to getting it into the core. 2. It provides a useful marker class. MA could inherit from it (and use itself for it's data attribute) and then asanyarray would behave properly. MA could also use this, or a subclass, as the mask object preventing anyone from accidentally using it as data (they could always use it on purpose with asarray). 3. It provides a platform for people to build other, ndarray-like classes in Pure python. This is my main interest. I've put together a thin shell over numpy that strips it down to it's abolute essentials including a stripped down version of ndarray that removes most of the methods. All of the __array_wrap__[1] stuff works quite well most of the time, but there's still some issues with being a subclass when this particular class is conceptually a superclass. If we had an array superclass of some sort, I believe that these would be resolved. In principle at least, this shouldn't be that hard. I think it should mostly be rearanging some code and adding some wrappers to existing functions. That's in principle. In practice, I'm not certain yet as I haven't investigated the code in question in much depth yet. I've been meaning to write this up into a more fleshed out proposal, but I got distracted by the whole Protocol discussion on python-dev3000. This writeup is pretty weak, but hopefully you get the idea. Anyway, this is somethig that I would be willing to put some time on that would benefit both me and probably the MA folks as well. Regards, -tim

On 4/7/06, Tim Hochberg wrote:

[...] Perhaps MaskedArray should inherit from ndarray for the time being. Many of the methods would need to reimplemented anyway, but it would make asanyarray work. Someone was just complaining about asarray munging his arrays. That's correct behaviour, but it would be nice if asanyarray did the right thing. I suppose we could just special case asanyarray to ignore MaskedArrays, that might be better since it's less constraining from an implementation side too.

Just for the record. Currently MA does not inherit from ndarray. There are some benefits to be gained from changing MA design from containment to inheritance, by I am very sceptical about the use of inheritance in the array setting.

...

This may be an oportune time to propose something that's been cooking in the back of my head for a week or so now: A stripped down array superclass.

This is a very worthwhile idea and I hate to see it burried in a non-descriptive thread. I've copied your proposal to the wiki at http://projects.scipy.org/scipy/numpy/wiki/ArraySuperClass.

On 4/7/06, Tim Hochberg wrote:

[...] Still it would be nice to have asanyarray pass masked arrays through somehow. I haven't thought this through very well, but I wonder if it would make sense for asanyarray to pass any object that supplies __array__. I'm leary of special casing asanyarray just for MA; somehow that seems the wrong approach.

One possiblility is to make asanyarray pass through objects that have __array_wrap__ attribute.

...

2. It should return masked array (with trivial mask), not ndarray.

So, just with mask = False? In a follow on message Pierre disagress and claims that what you really want is the ndarray since not everything will accept. Then I guess you'd need to call b.filled(fill).data. I agree with Sasha in principle but Pierre, perhaps in practice.

Well, if 'mask' became a default argument of ndarray, that wouldn't be a pb any longer. I'm quite for that.

I'm almost suggested it get renames a.asndarray(fill), except that asXXX has the wrong conotations. I think this one needs to bounce around some more.

tondarray(fill) ?

...

4. View/Copy inconsistency. Does not provide a method to fill values in-place. seems to work for this purpose. Can we just have filled return a copy?

Yes !

...

The problem with mask is that it's not clear whether True means the data is useful or unuseful.

I have to think twice all the time I want to create a mask that True means in fact that I don't want the data, whereas True selects the data for ndarray...

"hide" or "hidden"? A mask value of True essentially hides the underlying value. Unless when there's no underlying value ;). Rose, rose... I'm happy with mask, it reminds me of GRASS and gimp

The problem with the "mask" name is that ndarray already has unrelated "putmask" method. On the other hand putmask is redundant with fancy indexing. I have no other problem with "mask" name, so we may just decide to get rid of "putmask".

"putmask" really seems overkill indeed. I wouldn't miss it.

How do you set the mask? I keep getting attribute errors when I try it. And unmask would be a noop on an ndarray.

I've implemented something like that for some classes (inheriting from MA.MaskedArray). Never really used it yet, though #-------------------------------------------- def applymask(self,m): if not MA.is_mask(m): raise MA.MAError,"Invalid mask !" elif self._data.shape != m.shape: raise MA.MAError,"Mask and data not compatible." else: self._dmask = m

This may be an oportune time to propose something that's been cooking in the back of my head for a week or so now: A stripped down array superclass.

That'd be great indeed, and may solve some problems reported on th list about subclassing ndarray. AAMOF, I gave up trying to use ndarray as a superclass, and rely only on MA

Travis Oliphant wrote:

Pierre GM wrote:

...

...

decide to get rid of "putmask".

"putmask" really seems overkill indeed. I wouldn't miss it.

I'm not opposed to getting rid of putmask either. Several of the newer methods are open for discussion before 1.0. I'd have to check to be sure, but .take and .put are not entirely replaced by fancy-indexing. Also, fancy indexing has enough overhead that a method doing exactly what you want is faster.

I'm curious, what use cases does fancy indexing not handle that take works for? Not counting speed issues. Regards, -tim

Travis Oliphant wrote:

...

This may be an oportune time to propose something that's been cooking in the back of my head for a week or so now: A stripped down array superclass. The details of this are not at all locked down, but here's a strawman proposal.

This is in essence what I've been proposing since SciPy 2005. I want what goes into Python to be essentially just this super-class. Look at this http://numeric.scipy.org/array_interface.html

and check out this

svn co http://svn.scipy.org/svn/PEP arrayPEP

I've obviously been way over-booked to do this myself. Nick Coughlan expressed interest in this idea (he called it dimarray, but I like basearray better).

I'll look these over. I suppose I should have been paying more attention before!

...

We add an array superclass. call it basearray, that has the same C-structure as the existing ndarray. However, it has *no* methods or attributes.

Why not give it the attributes corresponding to it's C-structure. I'm happy with no methods though.

Mainly because I didn't want too much about whether a given method or attribute was a good idea and I was in a hurry when I tossed that proposal out. It seemed better to start with the most stripped down proposal I could come up and see what people demanded I add.. I'm actually sort of inclined to give it *read-only* attribute associated with C-structure, but no methods. That way you can examine the shape, type, etc but you can't set them [I'm specifically thinking of shape here, but there may be others].. I think that there are cases where you don't want the base array to be mutable at all, but I don't think introspection should be a problem. If the attributes were setabble, you could always override the them with readonly properties, but it'd be cleaner to just start with readonly functionality and add setability (is that a word?) only in those cases where it's needed.

...

1. If we're careful, this could be the basic array object that we propose, at least for the first roun,d for inclusion in the Python core. It's not useful for anything but passing data betwen various application that understand the data structure, but that in itself could be a huge win. And the fact that it's dirt simple would probably be an advantage to getting it into the core.

The only extra thing I'm proposing is to add the data-descriptor object into the Python core as well --- other-wise what do you do with PyArray_Descr * part of the C-structure?

Good point.

...

2. It provides a useful marker class. MA could inherit from it (and use itself for it's data attribute) and then asanyarray would behave properly. MA could also use this, or a subclass, as the mask object preventing anyone from accidentally using it as data (they could always use it on purpose with asarray).

...

3. It provides a platform for people to build other, ndarray-like classes in Pure python. This is my main interest. I've put together a thin shell over numpy that strips it down to it's abolute essentials including a stripped down version of ndarray that removes most of the methods. All of the __array_wrap__[1] stuff works quite well most of the time, but there's still some issues with being a subclass when this particular class is conceptually a superclass. If we had an array superclass of some sort, I believe that these would be resolved.

In principle at least, this shouldn't be that hard. I think it should mostly be rearanging some code and adding some wrappers to existing functions. That's in principle. In practice, I'm not certain yet as I haven't investigated the code in question in much depth yet. I've been meaning to write this up into a more fleshed out proposal, but I got distracted by the whole Protocol discussion on python-dev3000. This writeup is pretty weak, but hopefully you get the idea.

This is exactly what needs to be done to improve array-support in Python. This is the conclusion I came to and I'm glad to see that Tim is now basically having the same conclusion. There are obviously some details to work out. But, having a base structure to inherit from would be perfect.

Hmm. This idea seems to have a fair bit of consensus behind it. I guess that means I better looking into exactly what it would take to make it work. The details of what attributes to expose, etc are probably not too important to work out immediately. Regards, -tim

On 4/7/06, Pierre GM wrote:

... We're going towards MA as the default object.

I will be against changing the array structure to handle missing values. Let's keep the discussion focuced on the interface. Once we agree on the interface, it will be clear if any structural changes are necessary.

But then again, what would be the behavior to deal with missing values ?

We can postpone this discussion as well. Just add mask attribute that returns False and filled method that returns a copy is an example of a minimalistic change.

Using R-like na.actions ? That'd be great, but it's getting more complex.

I don't like na.actions. I think missing values should behave like IEEE NaNs and in the floating point case should be represented by NaNs. The functionality provided by na.actions can always be achieved by calling an extra function (filled or compress).

Oh, and another thing: if 'mask', or 'masked' becomes a default attribute of ndarrays, how do we define a mask? As a boolean ndarray whose 'mask' is always 'False' ? How do you __repr__ it ?

See above. For ndarray mask is always False unless an add-on module is loaded that redefines arithmetic to recognize special bit-patterns such as NaN or INT_MIN.

On 4/10/06, Bruce Southey wrote:

[...] I think the issue related to how masked values should be handled in computation. Does it matter if the result of an operation is due to a masked value or numerical problem (like dividing by zero)? (I am presuming that it is possible to identify this difference.) If not, then I support the idea of treating masked values as NaN.

IEEE standard prvides plenty of spare bits in NaNs to represent pretty much everything, and some languages take advantage of that feature. (I believe NA and NaN are distinct in R). In MA, however mask elements are boolean and no distinction is made between various reasons for not having a data element. For consistency, a non-trivial (not always false) implementation of ndarray.mask should return "not finite" and ignore bits that distinguish NaNs and infinities.

...

The functionality provided by na.actions can always be achieved by calling an extra function (filled or compress).

I am not clear on what you actually mean here. For example, if you are summing across a particular dimension, I would presume that any masked value would be ignored an that there would be some record of the fact that a masked value was encountered. This would allow that 'extra function' to handle the associated result. Alternatively the 'extra function' would have to be included as an argument - which is what the na.actions do.

If you sum along a particular dimension and encounter a masked value, the result is masked. The same is true if you encounter a NaN - the result is NaN. If you would like to ignore masked values, you write a.filled(0).sum() instead of a.sum(). In 1d case, you can also use a.compress().sum(). In other words, what in R you achieve with a flag, such as in sum(a, na.rm=TRUE), in numpy you achieve by an explicit call to "fill". This is not quite the same as na.actions in R, but that is what I had in mind.

On 4/10/06, Pierre GM wrote:

...

If you sum along a particular dimension and encounter a masked value, the result is masked.

That's not how it currently works (still on 0.9.6):

[... longish example snipped ...]

...

...

ma.array([1,1], mask=[0,1]).sum() 1

and frankly, I'd be quite frustrated if it had to change: - `filled` is not a ndarray method, which means that a.filled(0).sum() fails if a is not MA. Right now, I can use a.sum() without having to check the nature of a first.

This is exactly the point of the current discussion: make fill a method of ndarray. With the current behavior, how would you achieve masking (no fill) a.sum()?

- this behavior was already in Numeric

That's true, but it makes the result of sum(a) different from __builtins__.sum(a). I believe consistency with the python conventions is more important than with legacy Numeric in the long run.

[...]

- The current way reflects how mask are used in GIS or image processing.

Can you elaborate on this? Note that in R na.rm is false by default in sum:

sum(c(1,NA)) [1] NA

So it looks like the convention is different in the field of statistics.

...

If you would like to ignore masked values, you write a.filled(0).sum() instead of a.sum(). In 1d case, you can also use a.compress().sum().

Once again, Sasha, I'd agree with you if it wasn't a major difference

Array methods are a very recent addition to ma. We can still use this window of opportunity to get things right before to many people get used to the wrong behavior. (Note that I changed your implementation of cumsum and cumprod.)

...

In other words, what in R you achieve with a flag, such as in sum(a, na.rm=TRUE), in numpy you achieve by an explicit call to "fill". This is not quite the same as na.actions in R, but that is what I had in mind.

I kinda like the idea of a flag, though

With the flag approach making ndarray and ma.array interfaces consistent would require adding an extra argument to many methods. Instead, I poropose to add one method: fill to ndarray.

Pierre GM wrote:

...

...

[... longish example snipped ...]

...

...

ma.array([1,1], mask=[0,1]).sum()

1

So ? The result is not `masked`, the missing value has been omitted.

MA.array([[1,1],[1,1]],mask=[[0,1],[1,0]]).sum() array(data = [1 1], mask = [False False], fill_value=999999)

...

This is exactly the point of the current discussion: make fill a method of ndarray.

Mrf. I'm still not convinced, but I have nothing against it. Along with a mask=False_ by default ?

...

With the current behavior, how would you achieve masking (no fill) a.sum()?

Er, why would I want to get MA.masked along one axis if one value is masked ?

Any number of reasons I would think. It depends on what your using the data for. If the sum is the total amount that you spent in the month, and a masked value means you lost that check stub, then you don't know how much you actually spent and that value should be masked. To chose a boring example.

The current behavior is to mask only if all the values along that axis are masked:

MA.array([[1,1],[1,1]],mask=[[0,1],[1,1]]).sum() array(data = [1 999999], mask = [False True], fill_value=999999)

With a.filled(0).sum(), how would you distinguish between the cases (a) at least one value is not masked and (b) all values are masked ? (OK, by querying the mask with something in the line of a a._mask.all(axis), but it's longer... Oh well, I'll just to adapt)

Actually I'm going to ask you the same question. Why would care if all of the values are masked? I may be missing something, but either there's a sensible default value, in which case it doesn't matter how many values are masked, or you can't handle any masked values and the result should be masked if there are any masks in the input. Sasha's proposal handle those two cases well. Your behaviour a little more clunkily, but I'd like to understand why you want that behaviour. Regards, -tim

...

...

- this behavior was already in Numeric

That's true, but it makes the result of sum(a) different from __builtins__.sum(a). I believe consistency with the python conventions is more important than with legacy Numeric in the long run.

Array methods are a very recent addition to ma. We can still use this window of opportunity to get things right before to many people get used to the wrong behavior. (Note that I changed your implementation of cumsum and cumprod.)

Good points... We'll just have to put strong warnings everywhere.

...

...

- The current way reflects how mask are used in GIS or image processing.

Can you elaborate on this? Note that in R na.rm is false by default in sum:

...

sum(c(1,NA))

[1] NA

So it looks like the convention is different in the field of statistics.

MMh. *digs in his old GRASS scripts* OK, my bad. I had to fill missing values somehow, or at least check whether there were any before processing. I'll double check on that. Please temporarily forget that comment.

...

With the flag approach making ndarray and ma.array interfaces consistent would require adding an extra argument to many methods. Instead, I poropose to add one method: fill to ndarray.

OK, good point.

On a semantic aspect: While digging these GRASS scripts I mentioned, I realized/remembered that masked values are called 'null', when there's no data, a NAN, or just when you want to hide some values. What about 'null' instead of 'mask','missing','na' ?

------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Hi, My view is solely as user so I really do appreciate the thought that you all are putting into this! I am somewhat concerned that having to use filled() is an extra level of complexity and computational burden. For example, in computing the mean/average I using filled would require a one effort to get the sum and another to count the non-masked elements. For at least summation would it make more sense to add an optional flag(s) such that there appears little difference between a normal array and a masked array? For example, a.sum() is the current default a.sum(filled_value=x) where x is some value such as zero or other user defined value. a.sum(ignore_mask=True) or similar to address whether or not masked values should be used. I am also not clear on what happens with other operations or dimensions. Regards Bruce On 4/10/06, Pierre GM wrote:

...

[Sasha]

...

So ? The result is not `masked`, the missing value has been omitted. I am just making your point with a shorter example.

OK, now I get it :)

...

...

Er, why would I want to get MA.masked along one axis if one value is masked ?

[Tim] Any number of reasons I would think.

I understand that, and I eventually agree it should be the default.

...

[Sasha] Because if you don't know one of the addends you don't know the sum. Unless you want to discard some data on purpose.

...

Replacing missing values with zeros is not always the right strategy. If you know that your data has non-zero mean, for example, you might want to replace missing values with the mean instead of zero. Hence the need to get rid of filled_values

...

[Tim] Actually I'm going to ask you the same question. Why would care if all of the values are masked?

...

...

MA.array([[1,1],[1,1]],mask=[[0,1],[1,1]]).sum() array(data = [1 999999], mask = [False True], fill_value=999999)

[Sasha] I did not realize that, but it is really bad. What is the justification for this?

Masked values are not necessarily nans or missing. I quite regularly mask values that do not satisfy a given condition. For various reasons, I can't compress the array, I need to preserve its shape.

With the current behavior, a.sum() gives me the sum of the values that satisfy the condition. If there's no such value, the result is masked, and that way I know that the condition was never met. Here, I could use Sasha's method combined with a._mask.all, no problem

Another example: let x a 2D array with missing values, to be normalized along one axis. Currently, x/x.sum() give the result I want (provided it's true division). Sasha's method would give me a completely masked array.

...

...

Good points... We'll just have to put strong warnings everywhere. [Sasha] Do you agree with my proposal as long as we have explicit warnings in the documentation that methods behave differently from legacy functions?

Your points are quite valid. I'm just worried it's gonna break a lot of things in the next future. And where do we stop ? So, if we follow Sasha's way: x.prod() should be the same, right ? What about a.min(), a.max() ? a.mean() ?

------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Tim rant's a lot] Just to be clear, I'm not advocating getting rid of methods. I'm not advocating anything, that just seems to get me into trouble ;-) I still blame Java though. Regards, -tim

On 4/11/06, Sasha wrote:

On 4/10/06, Pierre GM wrote:

...

...

...

[... longish example snipped ...]

...

...

ma.array([1,1], mask=[0,1]).sum()

1 So ? The result is not `masked`, the missing value has been omitted.

I am just making your point with a shorter example.

...

[...] Mrf. I'm still not convinced, but I have nothing against it. Along with a mask=False_ by default ?

It looks like there is little opposition here. I'll submit a patch soon and unless better names are suggested, it will probably go in.

...

...

With the current behavior, how would you achieve masking (no fill) a.sum()? Er, why would I want to get MA.masked along one axis if one value is masked ?

Because if you don't know one of the addends you don't know the sum. Replacing missing values with zeros is not always the right strategy. If you know that your data has non-zero mean, for example, you might want to replace missing values with the mean instead of zero.

I feel that in general implicitly replacing masked values will definitely lead to bugs in my code. Unless it is really obvious what the best way to deal with the masked values is for the particular function, then I would definitely prefer to be explicit about it. In most cases there are a number of reasonable options for what can be done. Masking the result when masked values are involved seems the most transparent default option. For example, it gives me a really bad feeling to think that sum will automatically return the sum of all non-masked values. When dealing with large datasets, I will not always know when I need to be careful of missing values. Summing over the non-masked arrays will often not be the appropriate course and I fear that I will not notice that this has actually occurred. If masked values are returned it is pretty obvious what has happened and easily to go back and explicitly handle the masked data in another way if appropriate. Mike

In article , Sasha wrote:

I disagree. Numpy is pretty much alone among the array languages because it does not have "native" support for missing values. For the floating point types some rudimental support for nans exists, but is not really usable. There is no missing values machanism for integer types. I believe adding "filled" and maybe "mask" to ndarray (not necessarily under these names) could be a meaningful step towards "native" support for missing values.

I completely agree with this. I would really like to see proper native support for arrays with masked values in numpy (such that all ufuncs, functions, etc. work with masked arrays). I would be thrilled to be able to filter masked arrays, for instance. -- Russell