On Mon, Jun 27, 2011 at 8:18 PM, Matthew Brett <matthew.brett@gmail.com>wrote:

Hi,

On Mon, Jun 27, 2011 at 5:53 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

On Mon, Jun 27, 2011 at 9:55 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion

...

...

helped clarify the ideas for me. I've renamed and updated the NEP, then placed it into the master NumPy repository so it has a more permanent home here: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst In the NEP, I've tried to address everything that was raised in the original thread and in Nathaniel's followup 'Concepts' thread. To deal with the issue of whether a mask is True or False for a missing value, I've removed the 'mask' attribute entirely, except for ufunc-like functions np.ismissing and np.isavail which return the two styles of masks. Here's a high level summary of how I'm thinking of the topic, and what I will implement: Missing Data Abstraction There appear to be two useful ways to think about missing data that are worth supporting. 1) Unknown yet existing data 2) Data that doesn't exist In 1), an NA value causes outputs to become NA except in a small number of exceptions such as boolean logic, and in 2), operations treat the data as if there were a smaller array without the NA values. Temporarily Ignoring Data In some cases, it is useful to flag data as NA temporarily, possibly in several different ways, for particular calculations or testing out different ways of throwing away outliers. This is independent of the missing data abstraction, still requiring a choice of 1) or 2) above. Implementation Techniques There are two mechanisms generally used to implement missing data abstractions, 1) An NA bit pattern 2) A mask I've described a design in the NEP which can include both techniques using the same interface. The mask approach is strictly more general than the NA bit pattern approach, except for a few things like the idea of supporting the dtype 'NA[f8,InfNan]' which you can read about in the NEP. My intention is to implement the mask-based design, and possibly also implement the NA bit pattern design, but if anything gets cut it will be

...

...

NA bit patterns.

I have the impression that the mask-based design would be easier. Perhaps you could do that one first and folks could try out the API and see how

...

like it and discover whether the memory overhead is a problem in

has the they practice.

That seems like a risky strategy to me, as the most likely outcome is that people worried about memory will avoid masked arrays because they know they use more memory. The memory usage is predictable and we won't learn any more about it from use. We most of us already know if we're having to optimize code for memory.

You won't get complaints, you'll just lose a group of users, who will, I suspect, stick to NaNs, unsatisfactory as they are.

+1 - eat

See you,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi, On Mon, Jun 27, 2011 at 10:03 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett <matthew.brett@gmail.com> ...

...

That seems like a risky strategy to me, as the most likely outcome is that people worried about memory will avoid masked arrays because they know they use more memory.  The memory usage is predictable and we won't learn any more about it from use.  We most of us already know if we're having to optimize code for memory.

You won't get complaints, you'll just lose a group of users, who will, I suspect, stick to NaNs, unsatisfactory as they are.

This blade cuts both ways, we'd lose a group of users if we don't support masking semantics, too.

I didn't mean to agitate for my own use-case, I was only saying that 'implement masking, wait and see for na-dtype' was not a good strategy for deciding.

That said, Travis favors doing both, so there's a good chance there will be time for it.

That's very encouraging. I hope I can contribute somehow when the time comes, with review or some other way, Thanks a lot, Matthew

On Tue, Jun 28, 2011 at 9:06 AM, Nathaniel Smith <njs@pobox.com> wrote:

On Mon, Jun 27, 2011 at 2:03 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett <matthew.brett@gmail.com

wrote:

...

You won't get complaints, you'll just lose a group of users, who will, I suspect, stick to NaNs, unsatisfactory as they are.

This blade cuts both ways, we'd lose a group of users if we don't support masking semantics, too.

The problem is, that's inevitable. One might think that trying to find a compromise solution that picks a few key aspects of each approach would be a good way to make everyone happy, but in my experience, it mostly leads to systems that are a muddled mess and that make everyone unhappy. You're much better off saying screw it, these goals are in scope and those ones aren't, and we're going to build something consistent and powerful instead of focusing on how long the feature list is. That's also the problem with focusing too much on a list of use cases: you might capture everything on any single list, but there are actually an infinite variety of use cases that will arise in the future. If you can generalize beyond the use cases to find some simple and consistent mental model, and implement that, then that'll work for all those future use cases too. But sometimes that requires deciding what *not* to implement.

Just my opinion, but it's fairly hard won.

Anyway, it's pretty clear that in this particular case, there are two distinct features that different people want: the missing data feature, and the masked array feature. The more I think about it, the less I see how they can be combined into one dessert topping + floor wax solution. Here are three particular points where they seem to contradict each other:

Missing data: We think memory usage is critical. The ideal solution has zero overhead. If we can't get that, then at the very least we want the overhead to be 1 bit/item instead of 1 byte/item. Masked arrays: We say, it's critical to have good ways to manipulate the masking array, share it between multiple arrays, and so forth. And numpy already has great support for all those things! So obviously the masking array should be exposed as a standard ndarray.

Missing data: Once you've assigned NA to a value, you should *not* be able to get at what was stored there before. Masked arrays: You must be able to unmask a value and recover what was stored there before.

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there. They will get confused, and then store it away as another example of how computers are arbitrary and confusing and they're just too dumb to understand them, and I *hate* doing that to people. Plus the more that happens, the more they end up digging themselves into some hole by trying things at random, and then I have to dig them out again. So the point is, we can go either way, but in both ways there *is* a cost, and we have to decide.)

Missing data: It's critical that NAs propagate through reduction operations by default, though there should also be some way to turn this off. Masked arrays: Masked values should be silently ignored by reduction operations, and having to remember to pass a special flag to turn on this behavior on every single ufunc call would be a huge pain.

(Masked array advocates: please correct me if I'm misrepresenting you anywhere above!)

...

That said, Travis favors doing both, so there's a good chance there will be time for it.

One issue with the current draft is that I don't see any addressing of how masking-missing and bit-pattern-missing interact: a = np.zeros(10, dtype="NA[f8]") a.flags.hasmask = True a[5] = np.NA # Now what?

If you're going to implement both things anyway, and you need to figure out how they interact anyway, then why not split them up into two totally separate features?

Here's my proposal: 1) Add a purely dtype-based support for missing data: 1.A) Add some flags/metadata to the dtype structure to let it describe what a missing value looks like for an element of its type. Something like, an example NA value plus a function that can be called to identify NAs when they occur in arrays. (Notice that this interface is general enough to handle both the bit-stealing approach and the maybe() approach.) 1.B) Add an np.NA object, and teach the various coercion loops to use the above fields in the dtype structure to handle it. 1.C) Teach the various reduction loops that if a particular flag is set in the dtype, then they also should check for NAs and handle them appropriately. (If this flag is not set, then it means that this dtype's ufunc loops are already NA aware and the generic machinery is not needed unless skipmissing=True is given. This is useful for user-defined dtypes, and probably also a nice optimization for floats using NaN.) 1.D) Finally, as a convenience, add some standard NA-aware dtypes. Personally, I wouldn't bother with complicated string-based mini-language described in the current NEP; just define some standard NA-enabled dtype objects in the numpy namespace or provide a function that takes a dtype + a NA bit-pattern and spits out an NA-enabled dtype or whatever.

2) Add a better masked array support. 2.A) Masked arrays are simply arrays with an extra attribute '.visible', which is an arbitrary numpy array that is broadcastable to the same shape as the masked array. There's no magic here -- if you say a.visible = b.visible, then they now share a visibility array, according to the ordinary rules of Python assignment. (Well, there needs to be some check for shape compatibility, but that's not much magic.) 2.B) To minimize confusion with the missing value support, the way you mask/unmask items is through expressions like 'a.visible[10] = False'; there is no magic np.masked object. (There are a few options for what happens when you try to use scalar indexing explicitly to extract an invisible value -- you could return the actual value from behind the mask, or throw an error, or return a scalar masked array whose .visible attribute was a scalar array containing False. I don't know what the people who actually use this stuff would prefer :-).) 2.C) Indexing and shape-changing operations on the masked array are automatically applied to the .visible array as well. (Attempting to call .resize() on an array which is being used as the .visible attribute of some other array is an error.) 2.D) Ufuncs on masked arrays always ignore invisible items. We can probably share some code here between the handling of skipmissing=True for NA-enabled dtypes and invisible items in masked arrays, but that's purely an implementation detail.

This approach to masked arrays requires that the ufunc machinery have some special knowledge of what a masked array is, so masked arrays would have to become part of the core. I'm not sure whether or not they should be part of the np.ndarray base class or remain as a subclass, though. There's an argument that they're more of a convenience feature like np.matrix, and code which interfaces between ndarray's and C becomes more complicated if it has to be prepared to handle visibility. (Note that in contrast, ndarray's can already contain arbitrary user-defined dtypes, so the missing value support proposed here doesn't add any new issues to C interfacing.) So maybe it'd be better to leave it as a core supported subclass? Could go either way.

Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except that taking a masked view of an unmasked array allows ignoring values without destroying or copying the original data. The only downside I can see to an implementation using masks is memory and disk storage, and perhaps memory mapped arrays. And I rather expect the former to solve itself in a few years, eight gigs is becoming a baseline for workstations and in a couple of years I expect that to be up around 16-32, and a few years after that.... In any case we are talking 12% - 25% overhead, and in practice I expect it won't be quite as big a problem as folks project. Chuck

On 06/28/2011 07:26 AM, Nathaniel Smith wrote:

On Tue, Jun 28, 2011 at 9:38 AM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except that taking a masked view of an unmasked array allows ignoring values without destroying or copying the original data.

Charles, I know that :-).

But if that view thing is an advertised feature -- in fact, the key selling point for the masking-based implementation, included specifically to make a significant contingent of users happy -- then it's certainly user-visible. And it will make other users unhappy, like I said. That's life.

But who cares? My main point is that implementing a missing data solution and a separate masked array solution is probably less work than implementing a single everything-to-everybody solution *anyway*, *and* it might make both sets of users happier too. Notice that in my proposal, there's really nothing there that isn't already in Mark's NEP in some form or another, but in my version there's almost no overlap between the two features. That's not because I was trying to make them artificially different; it's because I tried to think of the most natural ways to satisfy each set of use cases, and they're just different.

I think you are exaggerating some of the differences associated with the implementation, and ignoring one *key* difference: for integer types, the masked implementation can handle the full numeric range of the type, while the bit-pattern approach cannot. Balanced against that, the *key* advantages of the bit-pattern approach would seem to be the simplicity of using a single array, particularly for IO (including memmapping) and interfacing with extension code. Although I am a heavy user of masked arrays, I consider these bit-pattern advantages to be substantial and deserving of careful consideration--perhaps of more weight and planning than they have gotten so far. Datasets on disk--e.g. climatological data, numerical model output, etc.--typically do use reserved values as missing value flags, although occasionally one also finds separate mask arrays. One of the real frustrations of the present masked array is that there is no savez/load support. I could roll my own by using a convention like saving the mask of xxx as xxx__mask__, and then reversing the process in a modified load; but I haven't gotten around to doing it. Regardless of internal implementation, I hope that core support for missing values will be included in savez/load. Eric

-- Nathaniel _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Jun 28, 2011, at 9:41 PM, Eric Firing wrote:

One of the real frustrations of the present masked array is that there is no savez/load support. I could roll my own by using a convention like saving the mask of xxx as xxx__mask__, and then reversing the process in a modified load; but I haven't gotten around to doing it. Regardless of internal implementation, I hope that core support for missing values will be included in savez/load.

Transform your masked array into a structured one (one field for the data, one for the mask), using the .torecords method, and use it as a regular ndarray.

On Tue, Jun 28, 2011 at 2:41 PM, Eric Firing <efiring@hawaii.edu> wrote:

...

On Tue, Jun 28, 2011 at 9:38 AM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except

On 06/28/2011 07:26 AM, Nathaniel Smith wrote: that

...

...

taking a masked view of an unmasked array allows ignoring values without destroying or copying the original data.

Charles, I know that :-).

But if that view thing is an advertised feature -- in fact, the key selling point for the masking-based implementation, included specifically to make a significant contingent of users happy -- then it's certainly user-visible. And it will make other users unhappy, like I said. That's life.

But who cares? My main point is that implementing a missing data solution and a separate masked array solution is probably less work than implementing a single everything-to-everybody solution *anyway*, *and* it might make both sets of users happier too. Notice that in my proposal, there's really nothing there that isn't already in Mark's NEP in some form or another, but in my version there's almost no overlap between the two features. That's not because I was trying to make them artificially different; it's because I tried to think of the most natural ways to satisfy each set of use cases, and they're just different.

I think you are exaggerating some of the differences associated with the implementation, and ignoring one *key* difference: for integer types, the masked implementation can handle the full numeric range of the type, while the bit-pattern approach cannot.

Balanced against that, the *key* advantages of the bit-pattern approach would seem to be the simplicity of using a single array, particularly for IO (including memmapping) and interfacing with extension code. Although I am a heavy user of masked arrays, I consider these bit-pattern advantages to be substantial and deserving of careful consideration--perhaps of more weight and planning than they have gotten so far.

Datasets on disk--e.g. climatological data, numerical model output, etc.--typically do use reserved values as missing value flags, although occasionally one also finds separate mask arrays.

One of the real frustrations of the present masked array is that there is no savez/load support. I could roll my own by using a convention like saving the mask of xxx as xxx__mask__, and then reversing the process in a modified load; but I haven't gotten around to doing it. Regardless of internal implementation, I hope that core support for missing values will be included in savez/load.

This sounds reasonable to me, and probably will require extending the file format a bit. -Mark

Eric

...

-- Nathaniel _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 06/28/2011 11:52 PM, Matthew Brett wrote:

Hi,

On Tue, Jun 28, 2011 at 5:38 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except that taking a masked view of an unmasked array allows ignoring values without destroying or copying the original data. The only downside I can see to an implementation using masks is memory and disk storage, and perhaps memory mapped arrays. And I rather expect the former to solve itself in a few years, eight gigs is becoming a baseline for workstations and in a couple of years I expect that to be up around 16-32, and a few years after that.... In any case we are talking 12% - 25% overhead, and in practice I expect it won't be quite as big a problem as folks project.

Or, in the case of 16 bit integers, 50% memory overhead.

I honestly find it hard to believe that I will not care about memory use in the near future, and I don't think it's wise to make decisions on that assumption.

In many sciences, waiting for the future makes things worse, not better, simply because the amount of available data easily grows at a faster rate than the amount of memory you can get per dollar :-) Dag Sverre

On 6/28/11 5:20 PM, Matthew Brett wrote:

Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith<njs@pobox.com> wrote: ...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it. Of course, that goes against common convention, but it might be easier to explain. Thanks, Jason

On Tue, Jun 28, 2011 at 6:00 PM, Matthew Brett <matthew.brett@gmail.com>wrote:

Hi,

On Tue, Jun 28, 2011 at 11:40 PM, Jason Grout <jason-sage@creativetrax.com> wrote:

...

On 6/28/11 5:20 PM, Matthew Brett wrote:

...

Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith<njs@pobox.com> wrote: ...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it. Of course, that goes against common convention, but it might be easier to explain.

I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion. For example, for:

a = np.array([np.NA, 1])

you might expect the result of a.sum() to be np.NA. That's what it is in R. However for:

b = np.array([np.IGNORE, 1])

you'd probably expect b.sum() to be 1. That's what it is for masked_array currently.

The current proposal fuses these two ideas with one implementation. Quoting from the NEP:

...

...

...

a = np.array([1., 3., np.NA, 7.], masked=True) np.sum(a) array(NA, dtype='<f8', masked=True) np.sum(a, skipna=True) 11.0

I agree with Nathaniel, that there is no practical way of avoiding the full 'NAs are in fact values where theres a False in the mask' concept, and that does impose a serious conceptual cost on the 'NA' user.

I'm not sure where the conceptual cost is coming from. If you're using missing values with the masked array implementation, all you see are missing value semantics. To see the additional masking behavior you have to deal with more than one view of the same data at the same time, something that is in and of itself already advanced for the novice user. -Mark

Best,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 29, 2011 at 8:20 AM, Lluís <xscript@gmx.net> wrote:

Matthew Brett writes:

...

...

Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it. Of course, that goes against common convention, but it might be easier to explain.

...

I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion.

This is how I see it:

...

...

...

a = np.array([0, 1, 2], dtype=int) a[0] = np.NA ValueError e = np.array([np.NA, 1, 2], dtype=int) ValueError b = np.array([np.NA, 1, 2], dtype=np.maybe(int)) m = np.array([np.NA, 1, 2], dtype=int, masked=True) bm = np.array([np.NA, 1, 2], dtype=np.maybe(int), masked=True) b[1] = np.NA np.sum(b) np.NA np.sum(b, skipna=True) 2 b.mask None m[1] = np.NA np.sum(m) 2 np.sum(m, skipna=True) 2 m.mask [False, False, True] bm[1] = np.NA np.sum(bm) 2 np.sum(bm, skipna=True) 2 bm.mask [False, False, True]

So:

* Mask takes precedence over bit pattern on element assignment. There's still the question of how to assign a bit pattern NA when the mask is active.

* When using mask, elements are automagically skipped.

* "m[1] = np.NA" is equivalent to "m.mask[1] = False"

* When using bit pattern + mask, it might make sense to have the initial values as bit-pattern NAs, instead of masked (i.e., "bm.mask == [True, False, True]" and "np.sum(bm) == np.NA")

There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false. Both NaN and Inf are implemented in hardware with the same idea as the NA bit pattern, but they do not follow NA missing value semantics. As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else. Everything on top of that is a choice of API and interface mechanisms. I want them to behave exactly the same except for that necessary difference, so that it will be possible to use the *exact same Python code* with either approach. Say you're using NA dtypes, and suddenly you think, "what if I temporarily treated these as NA too". Now you have to copy your whole array to avoid destroying your data! The NA bit pattern didn't save you memory here... Say you're using masks, and it turns out you didn't actually need masking semantics. If they're different, you now have to do lots of code changes to switch to NA dtypes! -Mark

Lluis

-- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi, On Wed, Jun 29, 2011 at 7:20 PM, Lluís <xscript@gmx.net> wrote:

Mark Wiebe writes:

...

There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false.

Well, my example contained a difference (the need for the "skipna=True" argument) precisely because it seemed that there was some need for different defaults.

Honestly, I think this difference breaks the POLA (principle of least astonishment).

[...]

...

As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else. Everything on top of that is a choice of API and interface mechanisms. I want them to behave exactly the same except for that necessary difference, so that it will be possible to use the *exact same Python code* with either approach.

I completely agree. What I'd suggest is a global and/or per-object "ndarray.flags.skipna" for people like me that just want to ignore these entries without caring about setting it on each operaion (or the other way around, depends on the default behaviour).

The downside is that it adds yet another tweaking knob, which is not desirable...

Oh - dear - that would be horrible, if, depending on the tweak somewhere in the distant past of your script, this:

...

...

a = np.array([np.NA, 1.0], masked=True) np.sum(a)

could return either np.NA or 1.0... Imagine someone twiddled the knob the other way and ran your script... See you, Matthew

Nathaniel Smith writes:

I know that the part 1 of that proposal would satisfy my needs, but I don't know as much about your use case, so I'm curious. Would that proposal (in particular, part 2, the classic masked-array part) work for you?

I'm for the option of having a single API when you want to have NA elements, regardless of whether it's using masks or bit patterns. My question is whether your ufuncs should react differently depending on the type of array you're using (bit pattern vs mask). In the beginning I thought it could make sense, as you know how you have created the array. So if you're using masks, you're probably going to ignore the NAs (becase you've explicitly set them, and you don't want a NA as the result of your summation). *But*, the more API/semantics both approaches share, the better; so I'd say that its better that they show the *very same* behaviour (w.r.t. "skipna"). My concern is now about how to set the "skipna" in a "comfortable" way, so that I don't have to set it again and again as ufunc arguments:

...

...

a array([NA, 2, 3]) b array([1, 2, NA]) a + b array([NA, 2, NA]) a.flags.skipna=True b.flags.skipna=True a + b array([1, 4, 3])

Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth

On Wed, Jun 29, 2011 at 5:42 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Wed, Jun 29, 2011 at 2:40 PM, Lluís <xscript@gmx.net> wrote:

...

I'm for the option of having a single API when you want to have NA elements, regardless of whether it's using masks or bit patterns.

I understand the desire to avoid having two different APIS...

[snip]

...

My concern is now about how to set the "skipna" in a "comfortable" way, so that I don't have to set it again and again as ufunc arguments:

...

...

...

a array([NA, 2, 3]) b array([1, 2, NA]) a + b array([NA, 2, NA]) a.flags.skipna=True b.flags.skipna=True a + b array([1, 4, 3])

...But... now you're introducing two different kinds of arrays with different APIs again? Ones where .skipna==True, and ones where .skipna==False?

I know that this way it's not keyed on the underlying storage format, but if we support both bit patterns and mask arrays at the implementation level, then the only way to make them have identical APIs is if we completely disallow unmasking, and shared masks, and so forth.

The right set of these conditions has been in the NEP from the beginning. Unmasking without value assignment is disallowed - the only way to "see behind the mask" or to share masks is with views. My impression is than more people are concerned with sharing the same data between different masks, something also supported through views. -Mark

Which doesn't seem like it'd be very popular (and would make including the mask-based implementation pretty pointless). So I think we have to assume that they will have APIs that are at least somewhat different. And then it seems like with this proposal then we'd actually end up with *4* different APIs that any particular array might follow... (or maybe more, depending on how arrays that had both a bit-pattern and mask ended up working).

That's why I was thinking the best solution might be to just bite the bullet and make the APIs *totally* different and non-overlapping, so it was always obvious which you were using and how they'd interact. But I don't know -- for my work I'd be happy to just pass skipna everywhere I needed it, and never unmask anything, and so forth, so maybe there's some reason why it's really important for the bit-pattern NA API to overlap more with the masked array API?

-- Nathaniel _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Mark Wiebe writes:

On Wed, Jun 29, 2011 at 1:20 PM, Lluís <xscript@gmx.net> wrote: [...]

...

As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else. Everything on top of that is a choice of API and interface mechanisms. I want them to behave exactly the same except for that necessary difference, so that it will be possible to use the *exact same Python code* with either approach.

I completely agree. What I'd suggest is a global and/or per-object "ndarray.flags.skipna" for people like me that just want to ignore these entries without caring about setting it on each operaion (or the other way around, depends on the default behaviour).

The downside is that it adds yet another tweaking knob, which is not desirable...

One way around this would be to create an ndarray subclass which changes that default. Currently this would not be possible to do nicely, but with the _numpy_ufunc_ idea I proposed in a separate thread a while back, this could work.

That does indeed sound good :) Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth

Oops, On Wed, Jun 29, 2011 at 8:32 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

Hi,

On Wed, Jun 29, 2011 at 6:22 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Wed, Jun 29, 2011 at 8:20 AM, Lluís <xscript@gmx.net> wrote:

...

Matthew Brett writes:

...

...

Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it.  Of course, that goes against common convention, but it might be easier to explain.

...

I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion.

This is how I see it:

...

...

...

a = np.array([0, 1, 2], dtype=int) a[0] = np.NA ValueError e = np.array([np.NA, 1, 2], dtype=int) ValueError b  = np.array([np.NA, 1, 2], dtype=np.maybe(int)) m  = np.array([np.NA, 1, 2], dtype=int, masked=True) bm = np.array([np.NA, 1, 2], dtype=np.maybe(int), masked=True) b[1] = np.NA np.sum(b) np.NA np.sum(b, skipna=True) 2 b.mask None m[1] = np.NA np.sum(m) 2 np.sum(m, skipna=True) 2 m.mask [False, False, True] bm[1] = np.NA np.sum(bm) 2 np.sum(bm, skipna=True) 2 bm.mask [False, False, True]

So:

* Mask takes precedence over bit pattern on element assignment. There's  still the question of how to assign a bit pattern NA when the mask is  active.

* When using mask, elements are automagically skipped.

* "m[1] = np.NA" is equivalent to "m.mask[1] = False"

* When using bit pattern + mask, it might make sense to have the initial  values as bit-pattern NAs, instead of masked (i.e., "bm.mask == [True,  False, True]" and "np.sum(bm) == np.NA")

There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false.

Well - first - it's helpful surely to separate the concepts and the implementation.

Concepts / use patterns (as delineated by Nathaniel): A) missing values == 'np.NA' in my emails.  Can we call that CMV (concept missing values)? B) masks == np.IGNORE in my emails . CMSK (concept masks)?

Implementations 1) bit-pattern == na-dtype - how about we call that IBP (implementation bit patten)? 2) array.mask.  IM (implementation mask)?

Nathaniel implied that:

CMV implies: sum([np.NA, 1]) == np.NA CMSK implies sum([np.NA, 1]) == 1

and indeed, that's how R and masked arrays respectively behave.  So I think it's reasonable to say that at least R thought that the bitmask implied the first and Pierre and others thought the mask meant the second.

The NEP as it stands thinks of CMV and and CM as being different views of the same thing,   Please correct me if I'm wrong.

...

Both NaN and Inf are implemented in hardware with the same idea as the NA bit pattern, but they do not follow NA missing value semantics.

Right - and that doesn't affect the argument, because the argument is about the concepts and not the implementation.

...

As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else.

I think Nathaniel's point was about the expected default behavior in the different concepts.

...

Everything on top of that is a choice of API and interface mechanisms. I want them to behave exactly the same except for that necessary difference, so that it will be possible to use the *exact same Python code* with either approach.

Right.  And Nathaniel's point is that that desire leads to fusion of the two ideas into one when they should be separated.  For example, if I understand correctly:

...

...

...

a = np.array([1.0, 2.0, 3, 7.0], masked=True) b = np.array([1.0, 2.0, np.NA, 7.0], dtype='NA[f8]') a[3] = np.NA  # actual real hand-on-heart assignment b[3] = np.NA # magic mask setting although it looks the same

I meant:

...

...

a = np.array([1.0, 2.0, 3.0, 7.0], masked=True) b = np.array([1.0, 2.0, 3.0, 7.0], dtype='NA[f8]') b[3] = np.NA  # actual real hand-on-heart assignment a[3] = np.NA # magic mask setting although it looks the same

Sorry, Matthew

Hi, On Wed, Jun 29, 2011 at 9:17 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

On Wed, Jun 29, 2011 at 1:32 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Wed, Jun 29, 2011 at 6:22 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Wed, Jun 29, 2011 at 8:20 AM, Lluís <xscript@gmx.net> wrote:

...

Matthew Brett writes:

...

...

Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it.  Of course, that goes against common convention, but it might be easier to explain.

...

I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion.

This is how I see it:

...

...

> a = np.array([0, 1, 2], dtype=int) > a[0] = np.NA ValueError > e = np.array([np.NA, 1, 2], dtype=int) ValueError > b  = np.array([np.NA, 1, 2], dtype=np.maybe(int)) > m  = np.array([np.NA, 1, 2], dtype=int, masked=True) > bm = np.array([np.NA, 1, 2], dtype=np.maybe(int), masked=True) > b[1] = np.NA > np.sum(b) np.NA > np.sum(b, skipna=True) 2 > b.mask None > m[1] = np.NA > np.sum(m) 2 > np.sum(m, skipna=True) 2 > m.mask [False, False, True] > bm[1] = np.NA > np.sum(bm) 2 > np.sum(bm, skipna=True) 2 > bm.mask [False, False, True]

So:

* Mask takes precedence over bit pattern on element assignment. There's  still the question of how to assign a bit pattern NA when the mask is  active.

* When using mask, elements are automagically skipped.

* "m[1] = np.NA" is equivalent to "m.mask[1] = False"

* When using bit pattern + mask, it might make sense to have the initial  values as bit-pattern NAs, instead of masked (i.e., "bm.mask == [True,  False, True]" and "np.sum(bm) == np.NA")

There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false.

Well - first - it's helpful surely to separate the concepts and the implementation.

Concepts / use patterns (as delineated by Nathaniel): A) missing values == 'np.NA' in my emails.  Can we call that CMV (concept missing values)? B) masks == np.IGNORE in my emails . CMSK (concept masks)?

Implementations 1) bit-pattern == na-dtype - how about we call that IBP (implementation bit patten)? 2) array.mask.  IM (implementation mask)?

Remember that the masks are invisible, you can't see them, they are an implementation detail. A good reason to hide the implementation is so it can be changed without impacting software that depends on the API.

It's not true that you can't see them because masks are using the same API as for missing values. Because they're using the same API, the person using the CMV stuff will soon find out about the masks, accidentally or not, then they will need to understand masking, and that is the problem we're discussing here. See you, Matthew

On Wed, Jun 29, 2011 at 4:21 PM, Eric Firing <efiring@hawaii.edu> wrote:

On 06/29/2011 09:32 AM, Matthew Brett wrote:

...

Hi,

[...]

...

Clearly there are some overlaps between what masked arrays are trying to achieve and what Rs NA mechanisms are trying to achieve. Are they really similar enough that they should function using the same API? And if so, won't that be confusing? I think that's the question that's being asked.

And I think the answer is "no". No more confusing to people coming from R to numpy than views already are--with or without the NEP--and not *requiring* people to use any NA-related functionality beyond what they are used to from R.

My understanding of the NEP is that it directly yields an API closely matching that of R, but with the opportunity, via views, to do more with less work, if one so desires. The present masked array module could be made more efficient if the NEP is implemented; regardless of whether this is done, the masked array module is not about to vanish, so anyone wanting precisely the masked array API will have it; and others remain free to ignore it (except for those of us involved in developing libraries such as matplotlib, which will have to support all variations of the new API along with the already-supported masked arrays).

In addition, for new code, the full-blown masked array module may not be needed. A convenience it adds, however, is the automatic masking of invalid values:

In [1]: np.ma.log(-1) Out[1]: masked

I'm sure this horrifies some, but there are times and places where it is a genuine convenience, and preferable to having to use a separate operation to replace nan or inf with NA or whatever it ends up being.

I added a mechanism to support this idea with the NA dtypes approach, spelled 'NA[f8,InfNan]'. Here, all Infs and NaNs are treated as NA by the system. -Mark If np.seterr were extended to allow such automatic masking as an option,

then the need for a separate masked array module would shrink further. I wouldn't mind having to use an explicit kwarg for ignoring NA in reduction methods.

Eric

...

See you,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 06/30/2011 08:53 AM, Nathaniel Smith wrote:

On Wed, Jun 29, 2011 at 2:21 PM, Eric Firing<efiring@hawaii.edu> wrote:

...

In addition, for new code, the full-blown masked array module may not be needed. A convenience it adds, however, is the automatic masking of invalid values:

In [1]: np.ma.log(-1) Out[1]: masked

I'm sure this horrifies some, but there are times and places where it is a genuine convenience, and preferable to having to use a separate operation to replace nan or inf with NA or whatever it ends up being.

Err, but what would this even get you? NA, NaN, and Inf basically all behave the same WRT floating point operations anyway, i.e., they all propagate?

Not exactly. First, it depends on np.seterr; second, calculations on NaN can be very slow, so are better avoided entirely; third, if an array is passed to extension code, it is much nicer if that code only has one NA value to handle, instead of having to check for all possible "bad" values.

Is the idea that if ufunc's gain a skipna=True flag, you'd also like to be able to turn it into a skipna_and_nan_and_inf=True flag?

No, it is to have a situation where skipna_and_nan_and_inf would not be needed, because an operation generating a nan or inf would turn those values into NA or IGNORE or whatever right away. Eric

-- Nathaniel _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 29, 2011 at 2:32 PM, Matthew Brett <matthew.brett@gmail.com>wrote:

Hi,

On Wed, Jun 29, 2011 at 6:22 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Wed, Jun 29, 2011 at 8:20 AM, Lluís <xscript@gmx.net> wrote:

...

Matthew Brett writes:

...

...

Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it.

Of

...

...

...

course, that goes against common convention, but it might be easier to explain.

...

I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion.

This is how I see it:

...

...

...

a = np.array([0, 1, 2], dtype=int) a[0] = np.NA ValueError e = np.array([np.NA, 1, 2], dtype=int) ValueError b = np.array([np.NA, 1, 2], dtype=np.maybe(int)) m = np.array([np.NA, 1, 2], dtype=int, masked=True) bm = np.array([np.NA, 1, 2], dtype=np.maybe(int), masked=True) b[1] = np.NA np.sum(b) np.NA np.sum(b, skipna=True) 2 b.mask None m[1] = np.NA np.sum(m) 2 np.sum(m, skipna=True) 2 m.mask [False, False, True] bm[1] = np.NA np.sum(bm) 2 np.sum(bm, skipna=True) 2 bm.mask [False, False, True]

So:

* Mask takes precedence over bit pattern on element assignment. There's still the question of how to assign a bit pattern NA when the mask is active.

* When using mask, elements are automagically skipped.

* "m[1] = np.NA" is equivalent to "m.mask[1] = False"

* When using bit pattern + mask, it might make sense to have the initial values as bit-pattern NAs, instead of masked (i.e., "bm.mask == [True, False, True]" and "np.sum(bm) == np.NA")

There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false.

Well - first - it's helpful surely to separate the concepts and the implementation.

Concepts / use patterns (as delineated by Nathaniel): A) missing values == 'np.NA' in my emails. Can we call that CMV (concept missing values)? B) masks == np.IGNORE in my emails . CMSK (concept masks)?

This is a different conceptual model than I'm proposing in the NEP. This is also exactly what I was trying to clarify in the first email in this thread under the headings "Missing Data Abstraction" and "Implementation Techniques". Masks are *just* an implementation technique. They imply nothing more, except through previously established conventions such as in various bitmasks, image masks, numpy.ma and others. masks != np.IGNORE bit patterns != np.NA Masks vs bit patterns and R's default NA vs rm.na NA semantics are completely independent, except where design choices are made that they should be related. I think they should be unrelated, masks and bit patterns are two approaches to solving the same problem.

Implementations 1) bit-pattern == na-dtype - how about we call that IBP (implementation bit patten)? 2) array.mask. IM (implementation mask)?

Nathaniel implied that:

CMV implies: sum([np.NA, 1]) == np.NA CMSK implies sum([np.NA, 1]) == 1

and indeed, that's how R and masked arrays respectively behave.

R and numpy.ma. If we're trying to be clear about our concepts and implementations, numpy.ma is just one possible implementation of masked arrays.

So I think it's reasonable to say that at least R thought that the bitmask implied the first and Pierre and others thought the mask meant the second.

R's model is based on years of experience and a model of what missing values implies, the bitmask implies nothing about the behavior of NA.

The NEP as it stands thinks of CMV and and CM as being different views of the same thing, Please correct me if I'm wrong.

...

Both NaN and Inf are implemented in hardware with the same idea as the NA bit pattern, but they do not follow NA missing value semantics.

Right - and that doesn't affect the argument, because the argument is about the concepts and not the implementation.

You just said R thought bitmasks implied something, and you're saying masked arrays imply something. If the argument is just about the missing value concepts, neither of these should be in the present discussion.

...

As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else.

I think Nathaniel's point was about the expected default behavior in the different concepts.

...

Everything on top of that is a choice of API and interface mechanisms. I want them to behave exactly the same except for that necessary difference, so that it will be possible to use the *exact same Python code* with either approach.

Right. And Nathaniel's point is that that desire leads to fusion of the two ideas into one when they should be separated. For example, if I understand correctly:

...

...

...

a = np.array([1.0, 2.0, 3, 7.0], masked=True) b = np.array([1.0, 2.0, np.NA, 7.0], dtype='NA[f8]') a[3] = np.NA # actual real hand-on-heart assignment b[3] = np.NA # magic mask setting although it looks the same

Why is one "magic" and the other "real"? All of this is already sitting on 100 layers of abstraction above electrons and atoms. If we're talking about "real," maybe we should be programming in machine code or using breadboards with individual transistors.

...

Say you're using NA dtypes, and suddenly you think, "what if I temporarily treated these as NA too". Now you have to copy your whole array to avoid destroying your data! The NA bit pattern didn't save you memory here... Say you're using masks, and it turns out you didn't actually need masking semantics. If they're different, you now have to do lots of code changes to switch to NA dtypes!

I personally have not run across that case. I'd imagine that, if you knew you wanted to do something so explicitly masking-like, you'd start with the masking interface.

People's use cases change over time, and sometimes one person's code is useful for others. I'd prefer to let people share. Clearly there are some overlaps between what masked arrays are trying

to achieve and what Rs NA mechanisms are trying to achieve. Are they really similar enough that they should function using the same API?

Yes.

And if so, won't that be confusing?

No, I don't believe so, any more than NA's in R, NaN's, or Inf's are already confusing. -Mark

I think that's the question that's being asked.

See you,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Thu, Jun 30, 2011 at 11:04 AM, Gary Strangman <strang@nmr.mgh.harvard.edu

wrote:

Clearly there are some overlaps between what masked arrays are

...

trying to achieve and what Rs NA mechanisms are trying to achieve. Are they really similar enough that they should function using the same API?

Yes.

And if so, won't that be confusing?

No, I don't believe so, any more than NA's in R, NaN's, or Inf's are already confusing.

As one who's been silently following (most of) this thread, and a heavy R and numpy user, perhaps I should chime in briefly here with a use case. I more-or-less always work with partially masked data, like Matthew, but not numpy masked arrays because the memory overhead is prohibitive. And, sad to say, my experiments don't always go perfectly. I therefore have arrays in which there is /both/ (1) data that is simply missing (np.NA?)--it never had a value and never will--as well as simultaneously (2) data that that is temporarily masked (np.IGNORE? np.MASKED?) where I want to mask/unmask different portions for different purposes/analyses. I consider these two separate, completely independent issues and I unfortunately currently have to kluge a lot to handle this.

Concretely, consider a list of 100,000 observations (rows), with 12 measures per observation-row (a 100,000 x 12 array). Every now and then, sprinkled throughout this array, I have missing values (someone didn't answer a question, or a computer failed to record a response, or whatever). For some analyses I want to mask the whole row (e.g., complete-case analysis), leaving me with array entries that should be tagged with all 4 possible labels:

1) not masked, not missing 2) masked, not missing 3) not masked, missing 4) masked, missing

Obviously #4 is "overkill" ... but only until I want to unmask that row. At that point, I need to be sure that missing values remain missing when unmasked. Can a single API really handle this?

The single API does support a masked array with an NA dtype, and the behavior in this case will be that the value is considered NA if either it is masked or the value is the NA bit pattern. So you could add a mask to an array with an NA dtype to temporarily treat the data as if more values were missing. One important reason I'm doing it this way is so that each NumPy algorithm and any 3rd party code only needs to be updated once to support both forms of missing data. The C API with masks is also a lot cleaner to work with than one for NA dtypes with the ability to have different NA bit patterns. -Mark

-best Gary

The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/**complianceline<http://www.partners.org/complianceline>. If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Thu, Jun 30, 2011 at 11:42 AM, Matthew Brett <matthew.brett@gmail.com>wrote:

Hi,

On Thu, Jun 30, 2011 at 5:13 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Thu, Jun 30, 2011 at 11:04 AM, Gary Strangman <strang@nmr.mgh.harvard.edu> wrote:

...

...

Clearly there are some overlaps between what masked arrays are trying to achieve and what Rs NA mechanisms are trying to achieve. Are they really similar enough that they should function using the same API?

Yes.

And if so, won't that be confusing?

No, I don't believe so, any more than NA's in R, NaN's, or Inf's are already confusing.

As one who's been silently following (most of) this thread, and a heavy

R

...

and numpy user, perhaps I should chime in briefly here with a use case. I more-or-less always work with partially masked data, like Matthew, but not numpy masked arrays because the memory overhead is prohibitive. And, sad to say, my experiments don't always go perfectly. I therefore have arrays in which there is /both/ (1) data that is simply missing (np.NA?)--it never had a value and never will--as well as simultaneously (2) data that that is temporarily masked (np.IGNORE? np.MASKED?) where I want to mask/unmask different portions for different purposes/analyses. I consider these two separate, completely independent issues and I unfortunately currently have to kluge a lot to handle this.

Concretely, consider a list of 100,000 observations (rows), with 12 measures per observation-row (a 100,000 x 12 array). Every now and then, sprinkled throughout this array, I have missing values (someone didn't answer a question, or a computer failed to record a response, or whatever). For some analyses I want to mask the whole row (e.g., complete-case analysis), leaving me with array entries that should be tagged with all 4 possible labels:

1) not masked, not missing 2) masked, not missing 3) not masked, missing 4) masked, missing

Obviously #4 is "overkill" ... but only until I want to unmask that row. At that point, I need to be sure that missing values remain missing when unmasked. Can a single API really handle this?

The single API does support a masked array with an NA dtype, and the behavior in this case will be that the value is considered NA if either it is masked or the value is the NA bit pattern. So you could add a mask to an array with an NA dtype to temporarily treat the data as if more values were missing.

Right - but I think the separated API is cleaner and easier to explain. Do you disagree?

Kind of, yeah. I think the important things to understand from the Python perspective are that there are two ways of doing missing values with NA that look exactly the same except for how you create the arrays. Since you know that the mask way takes more memory, and that's important for your application, you can decide to use the NA dtype without any additional depth. Understanding that one of them has a special signal for NA while the other uses masks in the background probably isn't even that important to understand to be able to use it. I bet lots of people who use R regularly couldn't come up with a correct explanation of how it works there. If someone doesn't understand masks, they can use their intuition based on the special signal idea without any difficulty. The idea that you can temporarily make some values NA without overwriting your data may not be intuitive at first glance, but I expect people will find it useful even if they don't fully understand the subtle details of the masking mechanism.

...

and any 3rd party code only needs to be updated once to support both

One important reason I'm doing it this way is so that each NumPy algorithm forms

...

of missing data.

Could you explain what you mean? Maybe a couple of examples?

Yeah, I've started adding some implementation notes to the NEP. First I need volunteers to review my current pull requests though. ;) -Mark

Whatever API results, it will surely be with us for a long time, and so it would be good to make sure we have the right one even if it costs a bit more to update current code.

Cheers,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Thu, Jun 30, 2011 at 11:54 AM, Lluís <xscript@gmx.net> wrote:

Mark Wiebe writes:

...

Why is one "magic" and the other "real"? All of this is already sitting on 100 layers of abstraction above electrons and atoms. If we're talking about "real," maybe we should be programming in machine code or using breadboards with individual transistors.

M-x butterfly RET

http://xkcd.com/378/

Ok, I've run this, how long does it take to execute? -Mark

-- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett <matthew.brett@gmail.com>wrote:

Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith <njs@pobox.com> wrote: ...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

My favorite way of explaining it would be to have a grid of numbers written on paper, then have several cardboards with holes poked in them in different configurations. Placing these cardboard masks in front of the grid would show different sets of non-missing data, without affecting the values stored on the paper behind them. -Mark

See you,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, Jun 28, 2011 at 6:57 PM, Pierre GM <pgmdevlist@gmail.com> wrote:

On Jun 29, 2011, at 1:39 AM, Mark Wiebe wrote:

...

On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett <matthew.brett@gmail.com> wrote: Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith <njs@pobox.com> wrote: ...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

My favorite way of explaining it would be to have a grid of numbers written on paper, then have several cardboards with holes poked in them in different configurations. Placing these cardboard masks in front of the grid would show different sets of non-missing data, without affecting the values stored on the paper behind them.

And when there's a hole (or just a blank) in your piece of paper ?

A hole means an unmasked element, no hole means a masked element. -Mark

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 06/29/2011 03:45 PM, Matthew Brett wrote:

Hi,

On Wed, Jun 29, 2011 at 12:39 AM, Mark Wiebe<mwwiebe@gmail.com> wrote:

...

On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett<matthew.brett@gmail.com> wrote:

...

Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith<njs@pobox.com> wrote: ...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

My favorite way of explaining it would be to have a grid of numbers written on paper, then have several cardboards with holes poked in them in different configurations. Placing these cardboard masks in front of the grid would show different sets of non-missing data, without affecting the values stored on the paper behind them.

Right - but here of course you are trying to explain the mask, and this is Nathaniel's point, that in order to explain NAs, you have to explain masks, and so, even at a basic level, the fusion of the two ideas is obvious, and already confusing. I mean this:

a[3] = np.NA

"Oh, so you just set the a[3] value to have some missing value code?"

"Ah - no - in fact what I did was set a associated mask in position a[3] so that you can't any longer see the previous value of a[3]"

"Huh. You mean I have a mask for every single value in order to be able to blank out a[3]? It looks like an assignment. I mean, it looks just like a[3] = 4. But I guess it isn't?"

"Er..."

I think Nathaniel's point is a very good one - these are separate ideas, np.NA and np.IGNORE, and a joint implementation is bound to draw them together in the mind of the user. Apart from anything else, the user has to know that, if they want a single NA value in an array, they have to add a mask size array.shape in bytes. They have to know then, that NA is implemented by masking, and then the 'NA for free by adding masking' idea breaks down and starts to feel like a kludge.

The counter argument is of course that, in time, the implementation of NA with masking will seem as obvious and intuitive, as, say, broadcasting, and that we are just reacting from lack of experience with the new API.

However, no matter how used we get to this, people coming from almost any other tool (in particular R) will keep think it is counter-intuitive. Why set up a major semantic incompatability that people then have to overcome in order to start using NumPy. I really don't see what's wrong with some more explicit API like a.mask[3] = True. "Explicit is better than implicit". Dag Sverre

On Wed, Jun 29, 2011 at 9:35 AM, Dag Sverre Seljebotn < d.s.seljebotn@astro.uio.no> wrote:

On 06/29/2011 03:45 PM, Matthew Brett wrote:

...

Hi,

On Wed, Jun 29, 2011 at 12:39 AM, Mark Wiebe<mwwiebe@gmail.com> wrote:

...

On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett<matthew.brett@gmail.com> wrote:

...

Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith<njs@pobox.com>

wrote:

...

...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

My favorite way of explaining it would be to have a grid of numbers written on paper, then have several cardboards with holes poked in them in different configurations. Placing these cardboard masks in front of the grid would show different sets of non-missing data, without affecting the values stored on the paper behind them.

Right - but here of course you are trying to explain the mask, and this is Nathaniel's point, that in order to explain NAs, you have to explain masks, and so, even at a basic level, the fusion of the two ideas is obvious, and already confusing. I mean this:

a[3] = np.NA

"Oh, so you just set the a[3] value to have some missing value code?"

"Ah - no - in fact what I did was set a associated mask in position a[3] so that you can't any longer see the previous value of a[3]"

"Huh. You mean I have a mask for every single value in order to be able to blank out a[3]? It looks like an assignment. I mean, it looks just like a[3] = 4. But I guess it isn't?"

"Er..."

I think Nathaniel's point is a very good one - these are separate ideas, np.NA and np.IGNORE, and a joint implementation is bound to draw them together in the mind of the user. Apart from anything else, the user has to know that, if they want a single NA value in an array, they have to add a mask size array.shape in bytes. They have to know then, that NA is implemented by masking, and then the 'NA for free by adding masking' idea breaks down and starts to feel like a kludge.

The counter argument is of course that, in time, the implementation of NA with masking will seem as obvious and intuitive, as, say, broadcasting, and that we are just reacting from lack of experience with the new API.

However, no matter how used we get to this, people coming from almost any other tool (in particular R) will keep think it is counter-intuitive. Why set up a major semantic incompatability that people then have to overcome in order to start using NumPy.

I'm not aware of a semantic incompatibility. I believe R doesn't support views like NumPy does, so the things you have to do to see masking semantics aren't even possible in R. I really don't see what's wrong with some more explicit API like

a.mask[3] = True. "Explicit is better than implicit".

I agree, but initial feedback was that the way R deals with NA values is very nice, and I've come to agree that it's worth emulating. -Mark

Dag Sverre _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 06/29/2011 07:38 PM, Mark Wiebe wrote:

...

On Wed, Jun 29, 2011 at 9:35 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no<mailto:d.s.seljebotn@astro.uio.no>> wrote:

On 06/29/2011 03:45 PM, Matthew Brett wrote: > Hi, > > On Wed, Jun 29, 2011 at 12:39 AM, Mark Wiebe<mwwiebe@gmail.com <mailto:mwwiebe@gmail.com>> wrote: >> On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett<matthew.brett@gmail.com<mailto:matthew.brett@gmail.com>> >> wrote: >>> >>> Hi, >>> >>> On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith<njs@pobox.com <mailto:njs@pobox.com>> wrote: >>> ... >>>> (You might think, what difference does it make if you *can* unmask an >>>> item? Us missing data folks could just ignore this feature. But: >>>> whatever we end up implementing is something that I will have to >>>> explain over and over to different people, most of them not >>>> particularly sophisticated programmers. And there's just no sensible >>>> way to explain this idea that if you store some particular value, then >>>> it replaces the old value, but if you store NA, then the old value is >>>> still there. >>> >>> Ouch - yes. No question, that is difficult to explain. Well, I >>> think the explanation might go like this: >>> >>> "Ah, yes, well, that's because in fact numpy records missing values by >>> using a 'mask'. So when you say `a[3] = np.NA', what you mean is, >>> 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`" >>> >>> Is that fair? >> >> My favorite way of explaining it would be to have a grid of numbers written >> on paper, then have several cardboards with holes poked in them in different >> configurations. Placing these cardboard masks in front of the grid would >> show different sets of non-missing data, without affecting the values stored >> on the paper behind them. > > Right - but here of course you are trying to explain the mask, and > this is Nathaniel's point, that in order to explain NAs, you have to > explain masks, and so, even at a basic level, the fusion of the two > ideas is obvious, and already confusing. I mean this: > > a[3] = np.NA > > "Oh, so you just set the a[3] value to have some missing value code?" > > "Ah - no - in fact what I did was set a associated mask in position > a[3] so that you can't any longer see the previous value of a[3]" > > "Huh. You mean I have a mask for every single value in order to be > able to blank out a[3]? It looks like an assignment. I mean, it > looks just like a[3] = 4. But I guess it isn't?" > > "Er..." > > I think Nathaniel's point is a very good one - these are separate > ideas, np.NA and np.IGNORE, and a joint implementation is bound to > draw them together in the mind of the user. Apart from anything > else, the user has to know that, if they want a single NA value in an > array, they have to add a mask size array.shape in bytes. They have > to know then, that NA is implemented by masking, and then the 'NA for > free by adding masking' idea breaks down and starts to feel like a > kludge. > > The counter argument is of course that, in time, the implementation of > NA with masking will seem as obvious and intuitive, as, say, > broadcasting, and that we are just reacting from lack of experience > with the new API.

However, no matter how used we get to this, people coming from almost any other tool (in particular R) will keep think it is counter-intuitive. Why set up a major semantic incompatability that people then have to overcome in order to start using NumPy.

I'm not aware of a semantic incompatibility. I believe R doesn't support views like NumPy does, so the things you have to do to see masking semantics aren't even possible in R. Well, whether the same feature is possible or not in R is irrelevant to whether a semantic incompatability would exist.

Views themselves are a *major* semantic incompatability, and are highly confusing at first to MATLAB/Fortran/R people. However they have major advantages outweighing the disadvantage of having to caution new users.

But there's simply no precedence anywhere for an assignment that doesn't erase the old value for a particular input value, and the advantages seem pretty minor (well, I think it is ugly in its own right, but that is besides the point...)

Dag Sverre _______________________________________________ Depending on what you really mean by 'precedence', in most stats software (R, SAS, etc.) it is completely up to the user to do this and do it correctly. Usually you store the original data and create new working data as needed as either the equivalent of a masked array or a view. Quite often I have need to have the same variable as a float and a string to be able to have the bets of both worlds (otherwise it is a

On 06/29/2011 01:07 PM, Dag Sverre Seljebotn wrote: pain of not only going back and forth but ensuring the right type is being used at the right time). The really really huge advantages of a masked arrays is that you only need the original data plus the mask and it is so easy to find the 'flagged' observations. There is actually no reason why you can not create a masked array or views in R - it is a language after all with the necessary features. So I do not see any semantic issues rather that it has not been implemented because there is insufficient interest. A likely reason is R's very strong statistical heritage so it has been up to the user to make their data fit the existing statistical routines. Really all this discussion just further highlights the value of masked arrays as it really extends the usage of some missing value coding. At present the cost is memory and the performance is a wait and see as there may not be a difference because in both cases the code has to know if a element is masked or missing and then how to deal with it. Bruce

On Wed, Jun 29, 2011 at 8:45 AM, Matthew Brett <matthew.brett@gmail.com>wrote:

Hi,

On Wed, Jun 29, 2011 at 12:39 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith <njs@pobox.com> wrote: ...

...

(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.

Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:

"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"

Is that fair?

My favorite way of explaining it would be to have a grid of numbers written on paper, then have several cardboards with holes poked in them in different configurations. Placing these cardboard masks in front of the grid would show different sets of non-missing data, without affecting the values stored on the paper behind them.

Right - but here of course you are trying to explain the mask, and this is Nathaniel's point, that in order to explain NAs, you have to explain masks, and so, even at a basic level, the fusion of the two ideas is obvious, and already confusing. I mean this:

a[3] = np.NA

"Oh, so you just set the a[3] value to have some missing value code?"

I would answer "Yes, that's basically true." The abstraction works that way, and there's no reason to confuse people with those implementation details right off the bat. When you introduce a new user to floating point numbers, it would seem odd to first point out that addition isn't associative. That kind of detail is important when you're learning more about the system and digging deeper. I think it was in a Knuth book that I read the idea that the best teaching is a series of lies that successively correct the previous lies.

"Ah - no - in fact what I did was set a associated mask in position a[3] so that you can't any longer see the previous value of a[3]"

"Huh. You mean I have a mask for every single value in order to be able to blank out a[3]? It looks like an assignment. I mean, it looks just like a[3] = 4. But I guess it isn't?"

"Er..."

I think Nathaniel's point is a very good one - these are separate ideas, np.NA and np.IGNORE, and a joint implementation is bound to draw them together in the mind of the user.

R jointly implements them with the rm.na=T parameter, and that's our model system for missing data.

Apart from anything else, the user has to know that, if they want a single NA value in an array, they have to add a mask size array.shape in bytes. They have to know then, that NA is implemented by masking, and then the 'NA for free by adding masking' idea breaks down and starts to feel like a kludge.

The counter argument is of course that, in time, the implementation of NA with masking will seem as obvious and intuitive, as, say, broadcasting, and that we are just reacting from lack of experience with the new API.

It will literally work the same as the implementation with NA dtypes, except for the masking semantics which requires the extra steps of taking views.

Of course, that does happen, but here, unless I am mistaken, the primary drive to fuse NA and masking is because of ease of implementation.

That's not the case, and I've tried to give a slightly better justification for this in my answer Lluis' email.

That doesn't necessarily mean that they don't go together - if something is easy to implement, sometimes it means it will also feel natural in use, but at least we might say that there is some risk of the implementation driving the API, and that that can lead to problems.

In the design process I'm doing, the implementation concerns are affecting the interface concerns and vice versa, but the missing data semantics are the main driver. -Mark

See you,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 6/27/11 9:53 AM, Charles R Harris wrote:

Some discussion of disk storage might also help. I don't see how the rules can be enforced if two files are used, one for the mask and another for the data, but that may just be something we need to live with.

It seems it wouldn't be too big deal to extend the *.npy format to include the mask. Could one memmap both the data array and the mask? Netcdf (and assume hdf) have ways to support masks as well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Mon, Jun 27, 2011 at 8:53 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

On Mon, Jun 27, 2011 at 12:44 PM, eat <e.antero.tammi@gmail.com> wrote:

...

Hi,

On Mon, Jun 27, 2011 at 6:55 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP, then placed it into the master NumPy repository so it has a more permanent home here:

https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst

In the NEP, I've tried to address everything that was raised in the original thread and in Nathaniel's followup 'Concepts' thread. To deal with the issue of whether a mask is True or False for a missing value, I've removed the 'mask' attribute entirely, except for ufunc-like functions np.ismissing and np.isavail which return the two styles of masks. Here's a high level summary of how I'm thinking of the topic, and what I will implement:

*Missing Data Abstraction*

There appear to be two useful ways to think about missing data that are worth supporting.

1) Unknown yet existing data 2) Data that doesn't exist

In 1), an NA value causes outputs to become NA except in a small number of exceptions such as boolean logic, and in 2), operations treat the data as if there were a smaller array without the NA values.

*Temporarily Ignoring Data* * * In some cases, it is useful to flag data as NA temporarily, possibly in several different ways, for particular calculations or testing out different ways of throwing away outliers. This is independent of the missing data abstraction, still requiring a choice of 1) or 2) above.

*Implementation Techniques* * * There are two mechanisms generally used to implement missing data abstractions, * * 1) An NA bit pattern 2) A mask

I've described a design in the NEP which can include both techniques using the same interface. The mask approach is strictly more general than the NA bit pattern approach, except for a few things like the idea of supporting the dtype 'NA[f8,InfNan]' which you can read about in the NEP.

My intention is to implement the mask-based design, and possibly also implement the NA bit pattern design, but if anything gets cut it will be the NA bit patterns.

Thanks again for all your input so far, and thanks in advance for your suggestions for improving this new revision of the NEP.

A very impressive PEP indeed.

Hi,

...

However, how would corner cases, like

...

...

...

a = np.array([np.NA, np.NA], dtype='f8', masked=True) np.mean(a, skipna=True)

This should be equivalent to removing all the NA values, then calling mean, like this:

...

...

...

b = np.array([], dtype='f8') np.mean(b) /home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2374: RuntimeWarning: invalid value encountered in double_scalars return mean(axis, dtype, out) nan

...

...

...

np.mean(a)

This would return NA, since NA values are sitting in positions that would affect the output result.

OK.

...

be handled?

My concern here is that there always seems to be such corner cases which can only be handled with specific context knowledge. Thus producing 100% generic code to handle 'missing data' is not doable.

Working out the corner cases for the functions that are already in numpy seems tractable to me, how to or whether to support missing data is something the author of each new function will have to consider when missing data support is in NumPy, but I don't think we can do more than provide the mechanisms for people to use.

Sure. I'll ride up with this and wait when I'll have some tangible to outperform the 'traditional' NaN handling. - eat

-Mark

...

Thanks, - eat

...

-Mark

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Mon, Jun 27, 2011 at 2:59 PM, <josef.pktd@gmail.com> wrote:

On Mon, Jun 27, 2011 at 2:24 PM, eat <e.antero.tammi@gmail.com> wrote:

...

On Mon, Jun 27, 2011 at 8:53 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Mon, Jun 27, 2011 at 12:44 PM, eat <e.antero.tammi@gmail.com> wrote:

...

Hi,

On Mon, Jun 27, 2011 at 6:55 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

First I'd like to thank everyone for all the feedback you're

...

...

...

...

clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP,

...

...

...

...

placed it into the master NumPy repository so it has a more permanent home here: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst In the NEP, I've tried to address everything that was raised in the original thread and in Nathaniel's followup 'Concepts' thread. To deal with the issue of whether a mask is True or False for a missing value, I've removed the 'mask' attribute entirely, except for ufunc-like functions np.ismissing and np.isavail which return the two styles of masks. Here's a high level summary of how I'm thinking of the topic, and what I will implement: Missing Data Abstraction There appear to be two useful ways to think about missing data that are worth supporting. 1) Unknown yet existing data 2) Data that doesn't exist In 1), an NA value causes outputs to become NA except in a small number of exceptions such as boolean logic, and in 2), operations treat the data as if there were a smaller array without the NA values. Temporarily Ignoring Data In some cases, it is useful to flag data as NA temporarily, possibly in several different ways, for particular calculations or testing out different ways of throwing away outliers. This is independent of the missing data abstraction, still requiring a choice of 1) or 2) above. Implementation Techniques There are two mechanisms generally used to implement missing data abstractions, 1) An NA bit pattern 2) A mask I've described a design in the NEP which can include both techniques using the same interface. The mask approach is strictly more general

...

...

...

...

the NA bit pattern approach, except for a few things like the idea of supporting the dtype 'NA[f8,InfNan]' which you can read about in the NEP. My intention is to implement the mask-based design, and possibly also implement the NA bit pattern design, but if anything gets cut it will be the NA bit patterns. Thanks again for all your input so far, and thanks in advance for your suggestions for improving this new revision of the NEP.

A very impressive PEP indeed.

Hi,

...

...

However, how would corner cases, like

...

...

> a = np.array([np.NA, np.NA], dtype='f8', masked=True) > np.mean(a, skipna=True)

This should be equivalent to removing all the NA values, then calling mean, like this:

...

...

...

b = np.array([], dtype='f8') np.mean(b)

/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2374:

...

RuntimeWarning: invalid value encountered in double_scalars return mean(axis, dtype, out) nan

...

...

...

> np.mean(a)

This would return NA, since NA values are sitting in positions that would affect the output result.

OK.

...

...

be handled? My concern here is that there always seems to be such corner cases

which

...

...

can only be handled with specific context knowledge. Thus producing 100% generic code to handle 'missing data' is not doable.

Working out the corner cases for the functions that are already in numpy seems tractable to me, how to or whether to support missing data is something the author of each new function will have to consider when missing data support is in NumPy, but I don't think we can do more than provide

providing, then than the

...

...

mechanisms for people to use.

Sure. I'll ride up with this and wait when I'll have some tangible to outperform the 'traditional' NaN handling. - eat

Just a question how things would work with the new model. How can you implement the "use" keyword from R's cov (or cor), with minimal data copying

I think the basic masked array version would (or does) just assign 0 to the missing values calculate the covariance or correlation and then correct with the correct count.

------------ cov(x, y = NULL, use = "everything", method = c("pearson", "kendall", "spearman"))

cor(x, y = NULL, use = "everything", method = c("pearson", "kendall", "spearman"))

cov2cor(V)

Arguments x a numeric vector, matrix or data frame. y NULL (default) or a vector, matrix or data frame with compatible dimensions to x. The default is equivalent to y = x (but more efficient). na.rm logical. Should missing values be removed?

use an optional character string giving a method for computing covariances in the presence of missing values. This must be (an abbreviation of) one of the strings "everything", "all.obs", "complete.obs", "na.or.complete", or "pairwise.complete.obs". ------------

especially I'm interested in the complete.obs (drop any rows that contains a NA) case

I think this is mainly a matter of extending NumPy's equivalent cov function with a parameter like this. Implemented in C, I'm sure it could be done with minimal copying, I'm not exactly sure how it will have to look implemented in Python. Perhaps someone could try it once I have a basic prototype ready for testing. -Mark

Josef

...

...

-Mark

...

Thanks, - eat

...

-Mark _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Jun 27, 2011, at 9:59 PM, josef.pktd@gmail.com wrote:

Just a question how things would work with the new model. How can you implement the "use" keyword from R's cov (or cor), with minimal data copying

I think the basic masked array version would (or does) just assign 0 to the missing values calculate the covariance or correlation and then correct with the correct count.

Basically, yes. Basic operations have a generic internal fill value (0 for sum/subtraction, 1 for multiplication/division), then you just have to correct by the count.

especially I'm interested in the complete.obs (drop any rows that contains a NA) case

In numpy.ma, there are functions to drop rows/columns that contain a masked value (they are in numpy.ma.extras, if I recall correctly): just filter your data by these functions before parsing it to np.cov. That's the kind of trivial example that is probably not worth overloading a function with optional parameters for.

On Mon, Jun 27, 2011 at 7:07 PM, Keith Goodman <kwgoodman@gmail.com> wrote:

...

First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP, then placed it into the master NumPy repository so it has a more permanent home here: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst In the NEP, I've tried to address everything that was raised in the original thread and in Nathaniel's followup 'Concepts' thread. To deal with the issue of whether a mask is True or False for a missing value, I've removed the 'mask' attribute entirely, except for ufunc-like functions np.ismissing and np.isavail which return the two styles of masks. Here's a high level

...

of how I'm thinking of the topic, and what I will implement: Missing Data Abstraction There appear to be two useful ways to think about missing data that are worth supporting. 1) Unknown yet existing data 2) Data that doesn't exist In 1), an NA value causes outputs to become NA except in a small number of exceptions such as boolean logic, and in 2), operations treat the data as if there were a smaller array without the NA values. Temporarily Ignoring Data In some cases, it is useful to flag data as NA temporarily, possibly in several different ways, for particular calculations or testing out different ways of throwing away outliers. This is independent of the missing data abstraction, still requiring a choice of 1) or 2) above. Implementation Techniques There are two mechanisms generally used to implement missing data abstractions, 1) An NA bit pattern 2) A mask I've described a design in the NEP which can include both techniques using the same interface. The mask approach is strictly more general than the NA bit pattern approach, except for a few things like the idea of supporting the dtype 'NA[f8,InfNan]' which you can read about in the NEP. My intention is to implement the mask-based design, and possibly also implement the NA bit pattern design, but if anything gets cut it will be

On Mon, Jun 27, 2011 at 8:55 AM, Mark Wiebe <mwwiebe@gmail.com> wrote: summary the

...

NA bit patterns. Thanks again for all your input so far, and thanks in advance for your suggestions for improving this new revision of the NEP.

I'm trying to understand this part of the missing data NEP:

"While numpy.NA works to mask values, it does not itself have a dtype. This means that returning the numpy.NA singleton from an operation like 'arr[0]' would be throwing away the dtype, which is still valuable to retain, so 'arr[0]' will return a zero-dimensional array either with its value masked, or containing the NA bit pattern for the array's dtype."

If I do something like this in Cython:

cdef np.float64_t ai for i in range(n): ai = a[i] ...

Then I need to specify the type of ai, say float64 as above.

What happens when a[i] is np.NA? Is ai still a float64? If NA is a bit pattern taken from float64 then a[i] could be float64, but if it is a 0d array then it would not be float64 and I assume I would run into problems or have to cast.

So what does all this mean for iterating over each element of an array in Cython or C? Would I need to check the mask of element i first and only assign to ai if the mask is True (meaning not missing)?

I'll have to add mention of Cython in the NEP. What should happen in Cython is the same thing that happens in Python, that the abstractions described in the NEP are followed precisely. Until the ability to work with missing values is added to Cython, the above will not be possible. The type np.float64_t isn't correct, Cython will need to add its own versions of np.nafloat64_t which it translates to/from by calling the appropriate NumPy APIs. -Mark

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, Jun 28, 2011 at 3:45 PM, Pierre GM <pgmdevlist@gmail.com> wrote:

All, I'm not sure I understand some aspects of Mark's new proposal, sorry (blame the lack of sleep). I'm pretty excited with the idea of built-in NA like np.dtype(NA['float64']), provided we can come with some shortcuts like np.nafloat64.

This could be created at NumPy startup time, no problem.

I think that would really take care of the missing data part in a consistent and non-ambiguous way. However, I understand that if a choice would be made, this approach would be dropped for the most generic "mask way", right ? (By "mask way", I mean something that is close (but actually optimized) to thenumpy.ma approach).

The NEP proposes strict NA missing value semantics, where the only way to get at the masked values is by having another view that doesn't have the value masked. If someone has use cases where this prevents some functionality they need, I'd love to hear them. So, taking this example

...

...

...

np.add(a, b, out=b, mask=(a > threshold)) If 'b' doesn't already have a mask, masked values will be lost if we go the mask way ? But kept if we go the bit way ? I prefer the latter, then Another advantage I see in the "bit-way' is that it's pretty close to the 'hardmask' idea. You'll never risk to lose the mask as it's already "burned" in the array...

I've nearly finished this parameter, and decided to call it 'where' instead, because it is operating like an SQL where clause. Here if neither a nor b are masked array it will only modify those values of b where the 'where' parameter has the value True. And now for something not that completely different:

* Would it be possible to store internally the addresses of the NAs only to save some space (in the metadata ?) and when the .mask or .valid property is called, to still get a boolean array with the same shape as the underlying array ?

Something like this could be possible, but would certainly complicate the implementation. If it were desired, it would be a follow-up feature. -Mark

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, Jun 28, 2011 at 6:55 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Tue, Jun 28, 2011 at 4:37 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

I've nearly finished this parameter, and decided to call it 'where' instead, because it is operating like an SQL where clause. Here if neither a nor b are masked array it will only modify those values of b where the 'where' parameter has the value True.

Probably a good idea, since in the current NEP, masks do not have a 'where' clause effect (at least for reduction operations), unless you pass skipna=True. So calling it mask= would be a bit confusing :-)

So if I understand right, f(a, b, where=c) is basically an optimized (copy-avoiding) version of f(a[c], b[c]) ?

Yes, of 'b[c] = f(a[c], b[c])'.

Except, in this case, c must be a boolean array of exactly the right shape, you don't support broadcasting or integer arrays or slices, etc.?

It supports broadcasting, but not arrays of indices. -Mark

-- Nathaniel _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, Jun 28, 2011 at 6:56 PM, Pierre GM <pgmdevlist@gmail.com> wrote:

On Jun 29, 2011, at 1:37 AM, Mark Wiebe wrote:

...

On Tue, Jun 28, 2011 at 3:45 PM, Pierre GM <pgmdevlist@gmail.com> wrote: ...

I think that would really take care of the missing data part in a consistent and non-ambiguous way. However, I understand that if a choice would be made, this approach would be dropped for the most generic "mask way", right ? (By "mask way", I mean something that is close (but actually optimized) to thenumpy.ma approach).

The NEP proposes strict NA missing value semantics, where the only way to get at the masked values is by having another view that doesn't have the value masked. If someone has use cases where this prevents some functionality they need, I'd love to hear them.

Mmh... Would you have an example ? I haven't caught up with my lack of sleep yet...

Sure, I'll copy one I made for the NEP for starters:

...

...

a = np.array([1,2]) b = a.view() b.flags.hasmask = True b array([1,2], masked=True) b[0] = np.NA b array([NA,2], masked=True) a array([1,2]) # The underlying number 1 value in 'a[0]' was untouched

At this point, there is no way to access the number 1 value using 'b', just using 'a'. If we assign to 'b[0]' it will also change a[0]:

...

...

b[0] = 3 a array([3,2])

...

So, taking this example

...

...

...

np.add(a, b, out=b, mask=(a > threshold)) If 'b' doesn't already have a mask, masked values will be lost if we go

the mask way ? But kept if we go the bit way ? I prefer the latter, then

...

Another advantage I see in the "bit-way' is that it's pretty close to the 'hardmask' idea. You'll never risk to lose the mask as it's already "burned" in the array...

I've nearly finished this parameter, and decided to call it 'where' instead, because it is operating like an SQL where clause. Here if neither a nor b are masked array it will only modify those values of b where the 'where' parameter has the value True.

OK, sounds fine. Pretty fine, actually. Just to be clear, if 'out' is not defined, the result is a masked array with 'where' as mask. What's the value below the mask ? np.NA ?

The value below the mask is like the result of np.empty(), and with strict missing value semantics, it shouldn't be possible to ever get at it. (except for breaking the rules from C code).

And now for something not that completely different:

...

* Would it be possible to store internally the addresses of the NAs only to save some space (in the metadata ?) and when the .mask or .valid property is called, to still get a boolean array with the same shape as the underlying array ?

Something like this could be possible, but would certainly complicate the implementation. If it were desired, it would be a follow-up feature.

Oh, no problem. I was suggesting a way to save some space, but if it's too tricky to implement, forget it.

Cool, I think the semantics with views of masks might not work either. With a bit-level mask the views would still be possible but more complicated than normal views. -Mark

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 29, 2011 at 2:26 AM, Dag Sverre Seljebotn < d.s.seljebotn@astro.uio.no> wrote:

On 06/27/2011 05:55 PM, Mark Wiebe wrote:

...

First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP, then placed it into the master NumPy repository so it has a more permanent home here:

https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst

One thing to think about is the presence of SSE/AVX instructions, which has the potential to change some of the memory/speed trade-offs here.

In the newest Intel-platform CPUs you can do 256-bit operations, translating to a theoretical factor 8 speedup for in-cache single precision data, and the instruction set is constructed for future expansion possibilites to 512 or 1024 bit registers.

The ufuncs themselves need a good bit of refactoring to be able to use these kinds of instructions well. I'm definitely thinking about this kind of thing while designing/implementing.

I feel one should take care to not design oneself into a corner where this can't (eventually) be leveraged.

1) The shuffle instructions takes a single byte as a control character for moving around data in different ways in 128-bit registers. One could probably implement fast IGNORE-style NA with a seperate mask using 1 byte per 16 bytes of data (with 4 or 8-byte elements). OTOH, I'm not sure if 1 byte per element kind of mask would be that fast (but I don't know much about this and haven't looked at the details).

This level of optimization, while important, is often dwarfed by the effects of cache. Because of the complexity of the system demanded by the functionality, I'm trying to favor simplicity and generality without precluding high performance. 2) The alternative "Parameterized Data Type Which Adds Additional Memory

for the NA Flag" would mean that contiguous arrays with NA's/IGNORE's would not be subject to vector instructions, or create a mess of copying in and out prior to operating on the data. This really seems like the worst of all possibilites to me.

This one was suggested on the list, so I added it. -Mark

(FWIW, my vote is in favour of both NA-using-NaN and IGNORE-using-explicit-masks, and keep the two as entirely seperate worlds to avoid confusion.)

Dag Sverre

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion