[Numpy-discussion] Changes to improve performance on small matricies

[Chris Barker]

Gary Bishop wrote:

> I'm writing to ask if these changes might make a worthy patch to NumPy?
> We have tested them with on Windows2k (both Native and under Cygwin)
> and on Linux. Soon, we'll have a test on Mac OS X.1.

I'd say yes, but we do have to be satisfied with the robustness of the
code, and I have no idea how to go about doing that. 

This brings up a good point. In Numeric.Py, the is the following
comment:

#The following functions are considered builtin, they all might be
#in C some day

I'd love to see that happen. As evidenced by this poster's example,
Numeric could be made a lot faster by optimizing a number of build-in
functions. What would it take to get a "Numeric Optimization project"
going??

A few ideas:

unit tests: If we had unit tests designed for all the built-in
functions, we would have a good way to know if a newly coded C version
is working right.

How-To_Docs: I have tried to write some Numeric extensions myself, and
it is very difficult. Writing one that does something unusual for my
particular application is not so hard with the current docs. In that
case it usually only has to work for one type, I know what shape the
array should be etc. Writing a universal extension to Numeric that would
work on any Numeric Array is a whole lot harder!! I am currently working
on two of these, and the hard parts for me are:

Doing something to all the data in a perhaps discontiguous array of
arbitrary shape. I have soemthing working now, but it's pretty slow.

Having a function work for all types of Arrays: dealing this this kin
dof dynamicism on C is very difficult for me.

The best place to look is clearly the source itself, but I have tried,
and it's pretty darn dense. A How-To would be pretty darn helpful.


I don't know who's going to write these things, but I think you could
enlarge the pool of NumPy developers quite a bit.

If anyone is interested, I have written "fastclip()" which does what
clip() does, but it does it in place, and only works on Floats (see
above) It is very fast for contiguous arrays, and disappointingly slow
on discontiguous arrays (also see above). Utill I get it working on all
types, there isn't much point in submitting it for inclusion in NumPy.

The other two I have written I use for dealing with binary data coming
in from files:

byteswap() does a bytswap in place, whioch is faster an more memory
efficient that the existing bytswapped() method.

changetype() changes the type of the array, using the same data:
changetype(m,typecode) = fromstring(m.tostring(),typecode)

It is a lot faster and memory efficient (the memory was the issue for
me, I'm dealing with up to 10MB arrays), but kind of an obscure need to,
so this may not be a candidate for inclusion in NumPy either. I still
have some questions about my code for that one: I'll post a separate
message about that.

-Chris
-- 
Christopher Barker,
Ph.D.                                                           
ChrisHBarker at home.net                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------