[Numpy-discussion] Explanation of some terms

[Travis Oliphant]

For the benefit of those who may be unfamiliar with ways to add new
functionality I will try to briefly summarize.  More information can be
found in the documentation and in the books that have been written about
Python.

There are two (three) ways to add a new object to Python:  using an
extension type and defining a class.  The fact that there are two
distinct ways to add new objects is often called the type-class dichotomy.
It is a goal of Py3K to somehow eliminate this distinction.  Another way
to add new behavior that I'll explain is to make the type an "extension
class."  Making the type a "subtype" of this fancy type gives a possible
direction for unifying types and classes.

Types
===============================

"Types" are more fundamental to the language and must be added using
compiled code (All of the types I've seen are in straight C since you
don't really buy anything by using C++ as Python itself is written in C).
You can investigate the type of an object from within python by using the
command type:

>>> type(a)  # prints the "type" of object a

There are many types defined in the Python core such as integers, floats, 
complex, lists, tuples, dictionaries, etc.


Python allows you to make new types.  These must be made in C (maybe C++
but again I don't think the extra complexity buys you anything since
Python is in C.)  A new type is a PyTypeObject basically filled with
function pointers and arrays of function pointers to handle the various
operations one might do on the new type.   This PyTypeObject is coupled
with a C-structure containing the "data" for the new type.  This data C
structure lists PyObject_HEAD as it's first member and then whatever other
data is necessary.  Making a new type is thus a matter of creating these
two C structures and filling in the TypeObject table with function
pointers to handle various operations (getting and setting attributes,
treating the type as an abstract number, sequence, or mapping, or printing
the object). 

Python has an abstract object interface on the C level, that is used, so
that if a type that has a "number" interface (operations) it can be used
like a number, if it has a "sequence" interface can be indexed like a
sequence, or if it has a "mapping" interface it can be indexed like a
dictionary.  


Classes
======================

A Python Class is at the C level just another "type."  There are actually
two "types" associated with a Python class:  an instance type and a class
type.  An instance of a class is the instance type.  So every instance of
any class has the same "type."  

What this means on a C level is that there is one more layer of
indirection for each "operation" in Python when the type is "class".  The
Python interpreter goes through the "class type" to see what to do and
finds the appropriate C function from that PyTypeObject Method table.
This C function does a dictionary lookup using the special method names
and executes the Python function associated with that name for the  
particular instance (which may call back into a compiled extension module
to do the actual work).   This level of indirection gives a great deal of
dynamic flexibility since classes can be subclassed and attributes can be
added dynamically, but there will be a performance hit which won't be
noticeable except inside Python iteration loops. 

So in reality there is no "type"-"class" dichotomy.  Everything is a
type.  It's just that classes are dynamic types which allow you to define
Python functions to implement the "method table"

The reason for the dichotomy is that classes are so useful, that people
really like them, and use them quite a bit so that the other static types
seem quite rigid in comparison.


Extension Classes
==================================

This is another fancy, dynamic "type" not distributed in the Python Core
but developed by Digital Creations (the Zope people) in order to let C
programmers "subclass" types.  I'm not an expert on these as I've never
really used them but as far as I can tell they bring the idea of
"dynamic types" to the C programmer.   This is accomplished by making all
types just subtypes of the extension class "type".  One way to
understand the result is by understanding what the type command
tells you about your new "extension class".  It will tell you
that's it's of type "extension class."  So, dynamic typing is again 
implemented with another layer of indirection where the fixed special C
functions of the extension class "type" call out to your particular set of 
registered C functions.  The difference is that the indirection is
all handled in C.  


So those are the choices for implementing new behavior in Python.  

Currently, Numerical Python is implemented as a new "type" which defines
all of these interfaces.  The mapping interface handles "extended
slicing," the "sequence" interface allows the array to return
something when len() is called for example, and the "number" interface
implements the operators. 

Actually, two new types are defined:  a "ufunc" type and an "array" type.
All of the operators are implemented as instances of the "ufunc" type.
The "ufunc" essentially encapsulates the "casting and broadcasting" rules
associated with elementwise operations.   The ufunc is not well-understood
by most non-developers I've talked too since most people don't instantiate
their own ufuncs (which must be instantiated in C).

The code works and is fast, but it can be hard to extend and there are
pieces that are poorly documented and hard to understand.   For example,
nobody has reworked the "extended slicing" syntax to enable
arbitrary-index slicing, despite many people who would like that feature
(actually I've heard that John Bernard did finally write some code to
do that but I've never seen it and it's not there now).

As mentioned before, David Ascher made the necessary changes to make
Numerical Python of type "extension class" which among other things,
allowed, the type to be "subclassed" from within Python.  I thought this
was a nice solution and we'd have to hear from him as to what went wrong.
The only trouble I had with it is that the C-API changed slightly in that
Arrays were no longer of type Array_Type and code that depended on it
would break (the same is true of any redesign making Python arrays a
class).  We'd have to hear from him as to what other problems he
saw.  It still doesn't solve the problem of maintainability of the C-code
base, but it definitely gave a more flexible result to the Python user.  

Perhaps retrofitting the ExtensionClass solution with an enhanced C-API
would be a better solution.  We really need David's input on that
suggestion...


The idea I've put forward is to make the object "classes" but I would
support the "extension class" solution as well.  Regardless of how it is
implemented, we still need to design the appropriate "objects"
(arraytype, NDArray, Ufunc) and how they interact with each other, as well
as a suitable C-API so that they work together seemlessly.  

I hope this helps some readers who are less familiar with extending
Python.  DISCLAIMER:  I am not the world's expert on these issues but I do
have some experience, so take what lessons you may.

Best wishes,

Travis Oliphant