[Python-Dev] Re: Numeric life as I see it

[Travis Oliphant]

> Martin v. Löwis wrote:
> The PEP should list the options, include criteria
>
>> for selection, and then propose a choice. People can then discuss
>> whether the list of options is complete (if not, you need to extend
>> it), whether the criteria are agreed (they might be not, and there
>> might be difficult consensus, which the PEP should point out), and
>> whether the choice is the right one given the criteria (there should
>> be no debate about this - everybody should agree factually that the
>> choice meets the criteria best).
>>
>
> Unrealistic. I think it is undisputed that there are people with 
> irreconcilably different needs. Frankly, we spent many, many months on 
> the design of Numeric and it represents a set of compromises already. 
> However, the one thing it wouldn't compromise on was speed, even at 
> the expense of safety. A community exists that cannot live with this 
> compromise. We were told that the Python core could also not live with 
> that compromise.


I'm not sure I agree.  The ufuncobject is the only place where this 
concern existed (should we trip OverFlow, ZeroDivision, etc. errors 
durring array math).   Numarray introduced and implemented the concept 
of error modes that can be pushed and popped.  I believe this is the 
right solution for the ufuncobject.

One question we are pursuing is could the arrayobject get into the core 
without a particular ufunc object.   Most see this as sub-optimal, but 
maybe it is the only way.

>
> Over the years there was pressure to add safety, convenience, 
> flexibility, etc., all sometimes incompatible with speed. Numarray 
> represents in some sense the set of compromises in that direction, 
> besides its technical innovations. Numeric / Numeric3 represents the 
> need for speed camp.

I don't see numarray as representing this at all.  To me, numarray 
represents the desire to have more flexible array types (specifically 
record arrays and maybe character arrays).   I personally don't see 
Numeric3 as in any kind of "need for speed" camp either.  I've never 
liked this distinction, because I don't think it represents a true 
dichotomy.  To me,  the differences between Numeric3 and numarray are 
currently more "architectural" than implementational.    

Perhaps you are referring to the fact that because numarray has several 
portions written in Python it is "more flexible" or "more convenient" 
but slower for small arrays.  If you are saying that then I guess 
Numeric3 is a "need for speed" implementation, and I apologize for not 
understanding. 

>
> I think it is reasonable to suppose that the need for speed piece can 
> be wrapped suitably by the need for safety-flexibility-convenience 
> facilities. I believe that hope underlies Travis' plan.

If the "safety-flexibility-convenience" facilities can be specified, 
then I'm all for one implementation.    Numeric3 design goals do not go 
against any of these ideas intentionally.

>
> The Nummies (the official set of developers) thought that the Numeric 
> code base was an unsuitable basis for further development. There was 
> no dissent about that at least. My idea was to get something like what 
> Travis is now doing done to replace it. I felt it important to get 
> myself out of the picture after five years as the lead developer 
> especially since my day job had ceased to involve using Numeric.

Some of the parts needed to be re-written, but I didn't think that meant 
moving away from the goal to have a single C-type that is the 
arrayobject.   During this process Python 2.2 came out and allowed 
sub-classing from C-types.  As Perry mentioned, and I think needs to be 
emphasized again, this changed things as any benefit from having a 
Python-class for the final basic array type disappeared --- beyond ease 
of prototyping and testing.

>
> However, removing my cork from the bottle released the unresolved 
> pressure between these two camps. My plan for transition failed. I 
> thought I had consensus on the goal and in fact it wasn't really 
> there. Everyone is perfectly good-willed and clever and trying hard to 
> "all just get along", but the goal was lost.  Eric Raymond should 
> write a book about it called "Bumbled Bazaar".

This is an accurate description.  Fortunately, I don't think any 
ill-will exists (assuming I haven't created any with my recent 
activities).  I do want to "get-along."  I just don't want to be silent 
when there are issues that I think I understand.

>
> I hope everyone will still try to achieve that goal. Interoperability 
> of all the Numeric-related software (including supporting a 'default' 
> plotting package) is required.

Utopia is always out of reach :-)

> Aside: While I am at it, let me reiterate what I have said to the 
> other developers privately: there is NO value to inheriting from the 
> array class. Don't try to achieve that capability if it costs 
> anything, even just effort, because it buys you nothing. Those of you 
> who keep remarking on this as if it would simply haven't thought it 
> through IMHO. It sounds so intellectually appealing that David Ascher 
> and I had a version of Numeric that almost did it before we realized 
> our folly.
>
I appreciate some of what Paul is saying here, but I'm not fully 
convinced that this is still true with Python 2.2 and up new-style 
c-types.   The concerns seem to be over the fact that you have to 
re-implement everything in the sub-class because the base-class will 
always return one of its objects instead of a sub-class object.
It seems to me, however,  that if the C methods use the object type 
alloc function when creating new objects then some of this problem is 
avoided (i.e. if the method is called with a sub-class type passed in, 
then a sub-class type gets set).

Have you looked at how Python now allows sub-classing in C?  I'm not an 
expert here, but it seems like a lot of the problems you were discussing 
have been ameliorated.  There are probably still issues, but....

I will know more when I seen what happens with a Matrix Object 
inheriting from a Python C-array object.

I'm wondering if anyone else with more knowledge about new-style c-types 
could help
here and show me the error of my thinking. 

-Travis