[Numpy-discussion] Medians that ignore values

[David Cournapeau]

Peter Saffrey wrote:
>
> I rejoiced when I saw this answer, because it looks like a function I can just
> drop in and it works. Unfortunately, nanmedian seems to be quite a bit slower
> than just using lists (ignoring nan values from my experiments) and a home-brew
> implementation of median. I was mostly using numpy for speed...

It may be that nanmedian is slow. But I would sincerly be surprised if
it were slower than python list, except for some pathological cases, or
maybe a bug in nanmedian. What do your data look like ? (size, number of
nan, etc...)

I quickly benchmarked on relatively small dataset (a few thousand
samples with a few random nan), and nanmedian is "only" a few times
slower than median.

>
> I would like to try the masked array approach, but the Ubuntu packages for scipy
> and matplotlib depend on numpy. Does anybody know whether I can naively do "sudo
> python setup.py install" on a more modern numpy without disturbing scipy and
> matplotlib, or do I need to uninstall all three packages and install them
> manually from source?

My advice would be to never ever install a package from source into
/usr. This will cause trouble. The way I do it is to install everything
from sources into $HOME/local (of course, any directory you have regular
write access to will do).

>
> On my 64 bit machine, the Ubuntu numpy package is even more out of date:
>
> $ dpkg -l | grep numpy
> ii  python-numpy                               1:1.0.4-6ubuntu3 
>
> Does anybody know why this is?

Yes, ubuntu updates every 6 months, the last time in last April. Numpy
1.1.0 (the first version after 1.0.4) was released in May. Also, Ubuntu
updates from debian, general 4-5 months before ubuntu release data. So
even if debian were to release a package the day we release a new
package, Ubuntu will be one year late.

I personally think that the solution would be to provide our own .deb up
to date, but this is a lot of work. I think Ondrej did some work related
to that; recent tools like opensuse build service and launchpad ppa
makes it somewhat a bit easier, too (for the build part, at least; you
still need to know how to build rpm/deb).

cheers,

David