[SciPy-User] Efficiently applying a function over a large array

[David Warde-Farley]

On 9-Feb-10, at 11:54 AM, Jose Gomez-Dans wrote:

> David, thanks for your reply
>
> On 9 February 2010 16:42, David Warde-Farley <dwf at cs.toronto.edu>  
> wrote:
>
> On 9-Feb-10, at 11:12 AM, Jose Gomez-Dans wrote:
>
> > for i in ny:
> >   for j in nx:
> >      Out[i,j] = MyFunc ( arr1[i,j,:], arr2[i,j,:], arr3[i,j,:] )
> >
> > but the array size is quite large (>1000x1000 elements), so I would
> > like to know what the most efficient way of doing this would be.
>
> You'll have to provide more details about what MyFunc is doing, as how
> efficient you can make it will depend critically on this.
>
> It is doing a fair bit of work, but in essence, from the three  
> vectors it gets, it generates a set of matrices, and performs a  
> number of matrix operations (sums and products), and a matrix  
> inversion too. What you get out of the function is a scalar.

The reason I ask is that depending on the specific operations it may  
be possible to write them as array-level operations on the entirety of  
arr1, arr2, arr3 without need for a loop/list comprehension/etc.

If you can't, you're stuck with a loop, but Cython would be an option  
for speeding it up if it turns out to be a bottleneck.

David