On Mon, Jun 27, 2011 at 5:53 PM, Charles R Harris
<
charlesr.harris@gmail.com> wrote:
>
>
> On Mon, Jun 27, 2011 at 9:55 AM, Mark Wiebe <
mwwiebe@gmail.com> wrote:
>>
>> First I'd like to thank everyone for all the feedback you're providing,
>> clearly this is an important topic to many people, and the discussion has
>> helped clarify the ideas for me. I've renamed and updated the NEP, then
>> placed it into the master NumPy repository so it has a more permanent home
>> here:
>>
https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> In the NEP, I've tried to address everything that was raised in the
>> original thread and in Nathaniel's followup 'Concepts' thread. To deal with
>> the issue of whether a mask is True or False for a missing value, I've
>> removed the 'mask' attribute entirely, except for ufunc-like functions
>> np.ismissing and np.isavail which return the two styles of masks. Here's a
>> high level summary of how I'm thinking of the topic, and what I will
>> implement:
>> Missing Data Abstraction
>> There appear to be two useful ways to think about missing data that are
>> worth supporting.
>> 1) Unknown yet existing data
>> 2) Data that doesn't exist
>> In 1), an NA value causes outputs to become NA except in a small number of
>> exceptions such as boolean logic, and in 2), operations treat the data as if
>> there were a smaller array without the NA values.
>> Temporarily Ignoring Data
>> In some cases, it is useful to flag data as NA temporarily, possibly in
>> several different ways, for particular calculations or testing out different
>> ways of throwing away outliers. This is independent of the missing data
>> abstraction, still requiring a choice of 1) or 2) above.
>> Implementation Techniques
>> There are two mechanisms generally used to implement missing data
>> abstractions,
>> 1) An NA bit pattern
>> 2) A mask
>> I've described a design in the NEP which can include both techniques using
>> the same interface. The mask approach is strictly more general than the NA
>> bit pattern approach, except for a few things like the idea of supporting
>> the dtype 'NA[f8,InfNan]' which you can read about in the NEP.
>> My intention is to implement the mask-based design, and possibly also
>> implement the NA bit pattern design, but if anything gets cut it will be the
>> NA bit patterns.
>
> I have the impression that the mask-based design would be easier. Perhaps
> you could do that one first and folks could try out the API and see how they
> like it and discover whether the memory overhead is a problem in practice.