There are two parts to this:

(1)  What is the new calculation for handling nan's?
(2)  What is the API for accessing the new calculation?

Before getting into the API (i.e. nanpdist vs. keyword vs. whatever),
I'd like better understand (1).

Here's a normal use of pdist (no nans):


In [158]: set_printoptions(precision=2)

In [159]: x = np.arange(1., 11).reshape(-1,2)

In [160]: x
Out[160]:
array([[  1.,   2.],
       [  3.,   4.],
       [  5.,   6.],
       [  7.,   8.],
       [  9.,  10.]])

In [161]: pdist(x)
Out[161]:
array([  2.83,   5.66,   8.49,  11.31,   2.83,   5.66,   8.49,   2.83,
         5.66,   2.83])


And here's how pdist currently handles nans:

In [162]: y = x.copy()

In [163]: y[0,1] = nan

In [164]: y[1,0] = nan

In [165]: y
Out[165]:
array([[  1.,  nan],
       [ nan,   4.],
       [  5.,   6.],
       [  7.,   8.],
       [  9.,  10.]])

In [166]: pdist(y)
Out[166]: array([  nan,   nan,   nan,   nan,   nan,   nan,   nan,  2.83,  5.66,  2.83])


That is, *any* distance involving a point that has a nan is nan.
This seems like a reasonable default behavior.

What should nanpdist(y) be?

Based on your code snippet on StackOverflow and your comment in the github
issue, my understanding is this: for any pair, you ignore the coordinates
where either has a nan (i.e. compute the distance in a lower dimension).
In this case, pdist(y) would be

    [nan, 4, 6, 8, 2, 4, 6, 2.83, 5.66, 2.83]

(I'm not sure if you would put nan or something else in that first position.)

Or, if we use the scaling of `n/(n - p)` that you suggested in the github issue,
where n is the dimension of the observations and p is the number of "missing"
coordinates,

    [nan, 8, 12, 16, 4, 8, 12, 2.83, 5.66, 2.83]

Is that correct?

What's the use-case for this behavior?  How widely used is it?


Warren