Oops, On Wed, Jun 29, 2011 at 8:32 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Wed, Jun 29, 2011 at 6:22 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Wed, Jun 29, 2011 at 8:20 AM, Lluís <xscript@gmx.net> wrote:
Matthew Brett writes:
Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it. Of course, that goes against common convention, but it might be easier to explain.
I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion.
This is how I see it:
a = np.array([0, 1, 2], dtype=int) a[0] = np.NA ValueError e = np.array([np.NA, 1, 2], dtype=int) ValueError b = np.array([np.NA, 1, 2], dtype=np.maybe(int)) m = np.array([np.NA, 1, 2], dtype=int, masked=True) bm = np.array([np.NA, 1, 2], dtype=np.maybe(int), masked=True) b[1] = np.NA np.sum(b) np.NA np.sum(b, skipna=True) 2 b.mask None m[1] = np.NA np.sum(m) 2 np.sum(m, skipna=True) 2 m.mask [False, False, True] bm[1] = np.NA np.sum(bm) 2 np.sum(bm, skipna=True) 2 bm.mask [False, False, True]
So:
* Mask takes precedence over bit pattern on element assignment. There's still the question of how to assign a bit pattern NA when the mask is active.
* When using mask, elements are automagically skipped.
* "m[1] = np.NA" is equivalent to "m.mask[1] = False"
* When using bit pattern + mask, it might make sense to have the initial values as bit-pattern NAs, instead of masked (i.e., "bm.mask == [True, False, True]" and "np.sum(bm) == np.NA")
There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false.
Well - first - it's helpful surely to separate the concepts and the implementation.
Concepts / use patterns (as delineated by Nathaniel): A) missing values == 'np.NA' in my emails. Can we call that CMV (concept missing values)? B) masks == np.IGNORE in my emails . CMSK (concept masks)?
Implementations 1) bit-pattern == na-dtype - how about we call that IBP (implementation bit patten)? 2) array.mask. IM (implementation mask)?
Nathaniel implied that:
CMV implies: sum([np.NA, 1]) == np.NA CMSK implies sum([np.NA, 1]) == 1
and indeed, that's how R and masked arrays respectively behave. So I think it's reasonable to say that at least R thought that the bitmask implied the first and Pierre and others thought the mask meant the second.
The NEP as it stands thinks of CMV and and CM as being different views of the same thing, Please correct me if I'm wrong.
Both NaN and Inf are implemented in hardware with the same idea as the NA bit pattern, but they do not follow NA missing value semantics.
Right - and that doesn't affect the argument, because the argument is about the concepts and not the implementation.
As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else.
I think Nathaniel's point was about the expected default behavior in the different concepts.
Everything on top of that is a choice of API and interface mechanisms. I want them to behave exactly the same except for that necessary difference, so that it will be possible to use the *exact same Python code* with either approach.
Right. And Nathaniel's point is that that desire leads to fusion of the two ideas into one when they should be separated. For example, if I understand correctly:
a = np.array([1.0, 2.0, 3, 7.0], masked=True) b = np.array([1.0, 2.0, np.NA, 7.0], dtype='NA[f8]') a[3] = np.NA # actual real hand-on-heart assignment b[3] = np.NA # magic mask setting although it looks the same
I meant:
a = np.array([1.0, 2.0, 3.0, 7.0], masked=True) b = np.array([1.0, 2.0, 3.0, 7.0], dtype='NA[f8]') b[3] = np.NA # actual real hand-on-heart assignment a[3] = np.NA # magic mask setting although it looks the same
Sorry, Matthew