[PYTHON MATRIX-SIG] Handling null data points

[Jim Fulton]

Konrad Hinsen wrote:
> 
> > I have two numeric arrays called A and B containing some real data.
> > Missing data has been replaced with a constant 'null' value of 100
> >
> >     A = (  20 10 100 6 )
> >     B = ( 100  1   2 1 )
> >
> > Now if I use Python to calculate A divided by B:
> >
> >     C = ( 0.2 10 50 6 )
> >
> > But when A or B is null I would like the answer to be null:
> >
> >     C = ( 100 10 100 6 )
> >
> 
> The cleanest solution to your problem would be to define a new
> data type, "number with special null value", and define all
> the arithmetic operations on it. You could even do that in Python,
> or if speed matters as a C extension. But even with a C extension
> you would not get the full speed of array operations, since your
> arrays would be of "general object" type.

The notion of a "missing value" is very important for data analysis.
I implemented such a type at USGS for use in database interfaces.
Unfortunately, this was never released and I haven't gotten around
to reimplementing it, although the implementation was close to
trivial. I think this would be very valuable.  If people are 
interested, I'd be willing to release an implementation.
I think it would also be valuable to consider supporting some 
special floating-point value (e.g. NAN) for use in floating-point
arrays.  This special value would be converted to/from the special
"missing value" object when accssed in Python.


-- 
Jim Fulton         Digital Creations
jim@digicool.com   540.371.6909
## Python is my favorite language ##
##     http://www.python.org/     ##

=================
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================