[Numpy-discussion] in the NA discussion, what can we agree on?
Nathaniel Smith
njs at pobox.com
Fri Nov 4 17:29:03 EDT 2011
On Fri, Nov 4, 2011 at 1:22 PM, T J <tjhnson at gmail.com> wrote:
> I agree that it would be ideal if the default were to skip IGNORED values,
> but that behavior seems inconsistent with its propagation properties (such
> as when adding arrays with IGNORED values). To illustrate, when we did
> "x+2", we were stating that:
>
> IGNORED(2) + 2 == IGNORED(4)
>
> which means that we propagated the IGNORED value. If we were to skip them
> by default, then we'd have:
>
> IGNORED(2) + 2 == 2
>
> To be consistent, then it seems we also should have had:
>
>>>> x + 2
> [3, 2, 5]
>
> which I think we can agree is not so desirable. What this seems to come
> down to is that we tend to want different behavior when we are doing
> reductions, and that for IGNORED data, we want it to propagate in every
> situation except for a reduction (where we want to skip over it).
>
> I don't know if there is a well-defined way to distinguish reductions from
> the other operations. Would it hold for generalized ufuncs? Would it hold
> for other functions which might return arrays instead of scalars?
Continuing my theme of looking for consensus first... there are
obviously a ton of ugly corners in here. But my impression is that at
least for some simple cases, it's clear what users want:
>>> a = [1, IGNORED(2), 3]
# array-with-ignored-values + unignored scalar only affects unignored values
>>> a + 2
[3, IGNORED(2), 5]
# reduction operations skip ignored values
>>> np.sum(a)
4
For example, Gary mentioned the common idiom of wanting to take an
array and subtract off its mean, and he wants to do that while leaving
the masked-out/ignored values unchanged. As long as the above cases
work the way I wrote, we will have
>>> np.mean(a)
2
>>> a -= np.mean(a)
>>> a
[-1, IGNORED(2), 1]
Which I'm pretty sure is the result that he wants. (Gary, is that
right?) Also numpy.ma follows these rules, so that's some additional
evidence that they're reasonable. (And I think part of the confusion
between Lluís and me was that these are the rules that I meant when I
said "non-propagating", but he understood that to mean something
else.)
So before we start exploring the whole vast space of possible ways to
handle masked-out data, does anyone see any reason to consider rules
that don't have, as a subset, the ones above? Do other rules have any
use cases or user demand? (I *love* playing with clever mathematics
and making things consistent, but there's not much point unless the
end result is something that people will use :-).)
-- Nathaniel
More information about the NumPy-Discussion
mailing list