Numeric3 and speed of new-style classes
I fixed a major oversight in Numeric3 and now it works pretty well (it is basically old Numeric with a bunch of new types, no math (yet)), using new-style classes, and some checks for alignment when dereferencing data. I did some simple array creation timing tests. An example of the results is given below. The bottom line is that new-style classes do nothing but speed up (a little bit) basic object creation. FYI: First test (Numeric3): .196081876755 import sys sys.path.insert(0,"../build/lib.linux-i686-2.3/") import time import multiarray start = time.time() for k in range(100000): g = multiarray.arraytype((4,3),'d') print time.time() - start Second test (Numeric): .203589916229 import Numeric import time start = time.time() for k in range(100000): g = Numeric.empty((4,3),'d') print time.time() - start
On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
I fixed a major oversight in Numeric3 and now it works pretty well (it is basically old Numeric with a bunch of new types, no math (yet)), using new-style classes, and some checks for alignment when dereferencing data.
I did some simple array creation timing tests. An example of the results is given below. The bottom line is that new-style classes do nothing but speed up (a little bit) basic object creation.
I think you've done a real service here, bounding the cost of new-style classes, but I'm not convinced we have a bottom line, at least not in the sense of my own FUD about new style classes. I'm not contesting the technical direction you're taking with Numeric3 (I hope it works, but the details are up to you), but I am contesting the idea that new style classes do nothing but speed up object creation. I think among the promises of new style classes are the ability to subclass in both C and Python, and my thinking is that if you actually do *that*, you pay for it through the nose. So, I'm trying to dispell the notion that new style classes are pure bliss; I think they need to be used either with insight (say a lot more than C++ OOP) or sidestepped altogether (as, IMHO, you have done). Using them, you're up against more than just faking OOP in C. I think one of the flaws currently in numarray is that I tried to push a Python class hierarchy down (or half-way down) into C. IMO, that's where some of the cost of new style classes would be incurred. The cost of the object __dict__ I mostly expected to be optimized out, perhaps by freelist magic or some lazy construction technique. If it's the latter. As far as state in the arrayobject goes, I *did* flatten the numarray class heirarchy. But I didn't flatten the numarray class hierarchy in terms of constructors and methods, and my sense is that there is a significant cost there. Calling up the class hierarchy uses Python method calls (maybe an error) which themselves require the construction, parsing, and destruction of tuple objects and at least some of the objects the tuples contain. So, I'd assert that unless you solve *that* problem, using new style classes and some form of inheritance, saying that new style classes do nothing but speed things up is stretching the truth. To really disprove the hidden costs in my FUD, I'd want to see a C basetype and inheritance by a Python subclass. If you actually use them for anything, I think you wind up paying Python function call overheads. That, anyway, is a problem to solve (or sidestep as you have done) for numarray. AFICT, among the things you're doing with Numeric3 (which I applaud) is basically throwing out inheritance from the fundamental array type; squashing all the constructor functionality for NDArray and NumArray into one type was more than I had the nerve to do. So, I know you're adding the ability to subclass, but you're not actually using it, so I think it's stretching the truth to say it speeds things up. In numarray, we use it, in both C and Python, and that's maybe not a good thing. If you can pull it off and still cleanly support the features of numarray, it will be a triumph of KISS. But that's not a given.
Please ignore that last post. It consists of unfinished thoughts which were sent accidentally. Doing numarray isn't easy. Regards, Todd On Fri, 2005-02-04 at 07:24, Todd Miller wrote:
On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
I fixed a major oversight in Numeric3 and now it works pretty well (it is basically old Numeric with a bunch of new types, no math (yet)), using new-style classes, and some checks for alignment when dereferencing data.
I did some simple array creation timing tests. An example of the results is given below. The bottom line is that new-style classes do nothing but speed up (a little bit) basic object creation.
I think you've done a real service here, bounding the cost of new-style classes, but I'm not convinced we have a bottom line, at least not in the sense of my own FUD about new style classes. I'm not contesting the technical direction you're taking with Numeric3 (I hope it works, but the details are up to you), but I am contesting the idea that new style classes do nothing but speed up object creation.
I think among the promises of new style classes are the ability to subclass in both C and Python, and my thinking is that if you actually do *that*, you pay for it through the nose. So, I'm trying to dispell the notion that new style classes are pure bliss; I think they need to be used either with insight (say a lot more than C++ OOP) or sidestepped altogether (as, IMHO, you have done). Using them, you're up against more than just faking OOP in C.
I think one of the flaws currently in numarray is that I tried to push a Python class hierarchy down (or half-way down) into C. IMO, that's where some of the cost of new style classes would be incurred. The cost of the object __dict__ I mostly expected to be optimized out, perhaps by freelist magic or some lazy construction technique. If it's the latter.
As far as state in the arrayobject goes, I *did* flatten the numarray class heirarchy. But I didn't flatten the numarray class hierarchy in terms of constructors and methods, and my sense is that there is a significant cost there. Calling up the class hierarchy uses Python method calls (maybe an error) which themselves require the construction, parsing, and destruction of tuple objects and at least some of the objects the tuples contain.
So, I'd assert that unless you solve *that* problem, using new style classes and some form of inheritance, saying that new style classes do nothing but speed things up is stretching the truth. To really disprove the hidden costs in my FUD, I'd want to see a C basetype and inheritance by a Python subclass.
If you actually use them for anything, I think you wind up paying Python function call overheads. That, anyway, is a problem to solve (or sidestep as you have done) for numarray.
AFICT, among the things you're doing with Numeric3 (which I applaud) is basically throwing out inheritance from the fundamental array type; squashing all the constructor functionality for NDArray and NumArray into one type was more than I had the nerve to do. So, I know you're adding the ability to subclass, but you're not actually using it, so I think it's stretching the truth to say it speeds things up.
In numarray, we use it, in both C and Python, and that's maybe not a good thing.
If you can pull it off and still cleanly support the features of numarray, it will be a triumph of KISS. But that's not a given.
------------------------------------------------------- This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion --
Todd Miller wrote:
Please ignore that last post. It consists of unfinished thoughts which were sent accidentally. Doing numarray isn't easy.
I think you've done a real service here, bounding the cost of new-style classes, but I'm not convinced we have a bottom line, at least not in the sense of my own FUD about new style classes. I'm not contesting the technical direction you're taking with Numeric3 (I hope it works, but the details are up to you), but I am contesting the idea that new style classes do nothing but speed up object creation.
Yes, your point about subclassing new-types causing speed slow downs is a very valid one and that has not been tested at all. But, for my
Unfinished or not, there were some good comments here, worth hearing. purposes, I was just interested to know if allowing a record Array to subclass from Numeric3 would cause slow-downs to the rest of Numeric. It appears that this will not be the case. Clearly any type that subclasses from Numeric3 might still have speed issues, but those were less of a concern for me, right now.
So, I'd assert that unless you solve *that* problem, using new style classes and some form of inheritance, saying that new style classes do nothing but speed things up is stretching the truth. To really disprove the hidden costs in my FUD, I'd want to see a C basetype and inheritance by a Python subclass.
Sure. But, my hope is that only RecordArrays would need to subclass in Python (or potentially in C). These might incur some creation "slowdown" but it would be inconsequential to the rest of Numeric.
If you can pull it off and still cleanly support the features of numarray, it will be a triumph of KISS. But that's not a given.
Clearly it's not a given, but I have high hopes. I really think it can be done.
-Travis
participants (2)
-
Todd Miller -
Travis Oliphant