Answers to Jaime's post:

Warren has already pointed this out, but let me insist: what is nanpdist, or the nan keyword expected to do? Treat pairs of vectors with NaNs as lower dimensional, removing pairs of entries where either is NaN? Do those results make any real sense? Thinking of euclidean distance for points in 3D space, I have trouble thinking of a practical situation where "if any Z coordinate is missing, just give me the distance of the projections onto the XY plane" would be anything but a misleading result. I presume the case is different for all those other distances I have never needed to use, so I am just curious of the use case.

Please see my answer to Warren about the use-case. In three dimensions this would certainly not make sense but my use-case has over three thousand dimensions. What I have in mind is a scaling factor for distance metrics, as suggested before, and an appropriate consideration of dissimilarity of the missing coordinate in similarity measures.
 

Looking at your linked post, from an implementation point of view, at the low level function that is actually going to do the heavy lifting, it is probable better to, rather than hardcode a check for NaN-ness, take a 'where' kwarg, as numpy ufuncs already do (http://docs.scipy.org/doc/numpy/reference/ufuncs.html#optional-keyword-arguments), and build the masking array in a higher level wrapper. This would make it easier to eventually make this functionality work with masked arrays or the like.

I'd be perfectly happy to do so. The hard-coded check is inspired by bottleneck which does exactly that for all its nan* functions. But I agree that a mask is preferable.
 

As a separate but related issue, I have had this PR open for almost a year now, https://github.com/scipy/scipy/pull/3163, and although me saying I want to complete it is getting old, hopefully whatever you have in mind can fit with the general structure of that.

I haven't fully grasped your code in umath_distance.c.src but that's probably a separate discussion. I also couldn't tell if some of that code is automatically generated or all written by hand.
 

Lastly, whatever you go for, I don't think you should do anything to pdist that you don't also do for cdist and the individual distance functions.


Noted and agreed.

Best,
Moritz