On Tue, Jun 28, 2011 at 6:00 PM, Matthew Brett <matthew.brett@gmail.com>wrote:
Hi,
On Tue, Jun 28, 2011 at 11:40 PM, Jason Grout <jason-sage@creativetrax.com> wrote:
On 6/28/11 5:20 PM, Matthew Brett wrote:
Hi,
On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith<njs@pobox.com> wrote: ...
(You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over and over to different people, most of them not particularly sophisticated programmers. And there's just no sensible way to explain this idea that if you store some particular value, then it replaces the old value, but if you store NA, then the old value is still there.
Ouch - yes. No question, that is difficult to explain. Well, I think the explanation might go like this:
"Ah, yes, well, that's because in fact numpy records missing values by using a 'mask'. So when you say `a[3] = np.NA', what you mean is, 'a._mask = np.ones(a.shape, np.dtype(bool); a._mask[3] = False`"
Is that fair?
Maybe instead of np.NA, we could say np.IGNORE, which sort of conveys the idea that the entry is still there, but we're just ignoring it. Of course, that goes against common convention, but it might be easier to explain.
I think Nathaniel's point is that np.IGNORE is a different idea than np.NA, and that is why joining the implementations can lead to conceptual confusion. For example, for:
a = np.array([np.NA, 1])
you might expect the result of a.sum() to be np.NA. That's what it is in R. However for:
b = np.array([np.IGNORE, 1])
you'd probably expect b.sum() to be 1. That's what it is for masked_array currently.
The current proposal fuses these two ideas with one implementation. Quoting from the NEP:
a = np.array([1., 3., np.NA, 7.], masked=True) np.sum(a) array(NA, dtype='<f8', masked=True) np.sum(a, skipna=True) 11.0
I agree with Nathaniel, that there is no practical way of avoiding the full 'NAs are in fact values where theres a False in the mask' concept, and that does impose a serious conceptual cost on the 'NA' user.
I'm not sure where the conceptual cost is coming from. If you're using missing values with the masked array implementation, all you see are missing value semantics. To see the additional masking behavior you have to deal with more than one view of the same data at the same time, something that is in and of itself already advanced for the novice user. -Mark
Best,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion