Assistance sought with mondo numerical app

[D. Michael McFarland]

Chris Barker <Chris.Barker at noaa.gov> writes:

> "D. Michael McFarland" wrote:
>> The bottlenecks now seem to be in
>> memory management, which I'm sure Python does better than I would in
>> C,
>
> better than me too, I'm sure, but NumPy code often ends up making a lot
> more copies of data that similar C code would, so that might very well
> get better in C too.

According to the profiler output, the single biggest time sink seems
to be the creation of new tensor objects, which are essentially NumPy
arrays with some bells and whistles.  I've always assumed (and
continue to) that NumPy was doing a good job of allocating memory and
copying array elements, and that I had simply chosen algorithms that
require the creation of too many new tensors.

>> and numerical linear algebra, which I'm sure Numeric does better
>> than I would.
>
> Are you using a ggod native BLAS? that can make a differnce too.

A good point, and something I've not looked at closely yet.  The
second bottleneck in the code is the inversion of rank 4 tensors, in
the course of which we do a singular value decomposition of a 9 x 9
NumPy array.  Speeding this up would certainly help overall.

One reason I'm reluctant to start rewriting my tensor lib (the good
parts of which are due to Konrad Hinsen, by the way) is that
eventually I'd want to call an external library for this and for a
linear equation solver.  Because that interface exists in NumPy, I've
felt I'd be reinventing the wheel, but maybe I should take another
look.  (In the long run, SciPy is probably how I want to do this, no?)

>> For now, though, the safest
>> route seems to be to throw CPU time at the existing code.
>
> I understand, and I may be able to offer a few cycles. Send me a
> private email.

Will do.  Thanks.

>> But first this guy needs to
>> graduate, for which he needs to crunch a _lot_ of numbers.  He's
>> swamped with related work, and I feel like I got him into this...
>
> Where would he be if he had tried to write all his code in Fortran
> or C??

You'd have to ask him for the full answer, but I remain convinced that
the savings in programming time and the clarity possible in Python are
well worth the run time they cost.  It's just that, dissertation
writing being what it is, things have come down to a crunch at the
end.  Our running joke is that in ten years' time we'll be running
these problems on our Palm XVII PDAs and laughing about it.

Thanks for your suggestions.  I'll take another look at how I might
tighten up some parts of the existing code.

Cheers,
Michael