[Numpy-discussion] boolean array indexing and broadcasting rules

[Mark Wiebe]

As part of supporting the NA mask, I've rewritten boolean indexing. Here's a
timing comparison of my version versus a previous version:

In [2]: np.__version__
Out[2]: '1.4.1'
In [3]: a = np.zeros((1000,1000))
In [4]: mask = np.random.rand(1000,1000) > 0.5
In [5]: timeit a[mask] = 1.5
10 loops, best of 3: 71.5 ms per loop

In [2]: np.__version__
Out[2]: '2.0.0.dev-a1e98d4'
In [3]: a = np.zeros((1000,1000))
In [4]: mask = np.random.rand(1000,1000) > 0.5
In [5]: timeit a[mask] = 1.5
100 loops, best of 3: 12.6 ms per loop

That's a 5.6 times speedup.

Unfortunately, it turns out that the old code didn't use NumPy broadcasting
rules. This change found a bug in the hardmask code of numpy.ma, so I think
there's definitely a case to be made that it is improvement, but at the same
time it looks like the NumPy and SciPy polynomial code is using this
behavior. I get test errors of the following form in Scipy:

======================================================================
ERROR: test_orthogonal_eval.TestPolys.test_sh_chebyu
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/case.py",
line 187, in runTest
    self.test(*self.arg)
  File
"/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/scipy/special/tests/test_orthogonal_eval.py",
line 111, in test_sh_chebyu
    param_ranges=[], x_range=[0, 1])
  File
"/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/scipy/special/tests/test_orthogonal_eval.py",
line 66, in check_poly
    ds.check()
  File
"/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/scipy/special/_testutils.py",
line 165, in check
    data = data[param_mask]
ValueError: operands could not be broadcast together with shapes (100,3)
(100)

What are people's thoughts on this predicament?

Cheers,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110729/b218a266/attachment.html>