[Numpy-discussion] Numeric3 and speed of new-style classes

[Todd Miller]

On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
> 
> I fixed a major oversight in Numeric3 and now it works pretty well (it 
> is basically old Numeric with a bunch of new types, no math (yet)), 
> using new-style classes, and some checks for alignment when 
> dereferencing data.
> 
> I did some simple array creation timing tests.  An example of the 
> results is given below.  The bottom line is that new-style classes do 
> nothing but speed up (a little bit) basic object creation. 

I think you've done a real service here,  bounding the cost of new-style
classes,  but I'm not convinced we have a bottom line,  at least not in
the sense of my own FUD about new style classes.  I'm not contesting the
technical direction you're taking with Numeric3 (I hope it works,  but
the details are up to you),  but I am contesting the idea that new style
classes do nothing but speed up object creation.

I think among the promises of new style classes are the ability to
subclass in both C and Python,  and my thinking is that if you actually
do *that*,  you pay for it through the nose.  So,  I'm trying to dispell
the notion that new style classes are pure bliss;  I think they need to
be used either with insight (say a lot more than C++ OOP) or sidestepped
altogether (as, IMHO, you have done).  Using them,  you're up against
more than just faking OOP in C.

I think one of the flaws currently in numarray is that I tried to push a
Python class hierarchy down (or half-way down) into C.   IMO,  that's
where some of the cost of new style classes would be incurred.  The cost
of the object __dict__ I mostly expected to be optimized out,  perhaps
by freelist magic or some lazy construction technique.  If it's the
latter.

As far as state in the arrayobject goes, I *did* flatten the numarray
class heirarchy.  But I didn't flatten the numarray class hierarchy in
terms of constructors and methods,  and my sense is that there is a
significant cost there.   Calling up the class hierarchy uses Python
method calls (maybe an error) which themselves require the construction,
parsing, and destruction of tuple objects and at least some of the
objects the tuples contain.  

So,  I'd assert that unless you solve *that* problem,  using new style
classes and some form of inheritance,  saying that new style classes do
nothing but speed things up is stretching the truth.  To really disprove
the hidden costs in my FUD,  I'd want to see a C basetype and
inheritance by a Python subclass.

If you actually use them for anything,  I think you wind up paying
Python function call overheads.  That, anyway,  is a problem to solve
(or sidestep as you have done) for numarray.

AFICT, among the things you're doing with Numeric3 (which I applaud) is
basically throwing out inheritance from the fundamental array type;
squashing all the constructor functionality for NDArray and NumArray
into one type was more than I had the nerve to do.  So, I know you're
adding the ability to subclass,  but you're not actually using it,  so I
think it's stretching the truth to say it speeds things up.

In numarray,  we use it,  in both C and Python,  and that's maybe not a
good thing.  

If you can pull it off and still cleanly support the features of
numarray,  it will be a triumph of KISS.  But that's not a given.