On March 7th Travis Oliphant and Perry Greenfield met Guido and Paul
Dubois to discuss some issues regarding the inclusion of an array
package within core Python.
The following represents thoughts and conclusions regarding our meeting
with Guido. They in no way represent the order of discussion with Guido
and some of the points we raise weren't actually mentioned during the
meeting, but instead were spurred by subsequent discussion after the
meeting with Guido.
1) Including an array package in the Python core. To start, before the
meeting we both agreed that we did not think that this itself was a
high priority in itself. Rather we both felt that the most important
issue was making arrays an acceptable and widely supported interchange
format (it may not be apparent to some that this does not require
arrays be in the core; more on that later). In discussing the
desirability of including arrays in the core with Guido, we quickly
came to the conclusion that not only was it not important, that in the
near term (the next couple years and possibly much longer) it was a bad
thing to do so. This is primarily because it would mean that updates to
the array package would wait on Python releases potentially delaying
important bug fixes, performance enhancements, or new capabilities
greatly. Neither of us envisions any scenario regarding array packages,
whether that be Numeric3 or numarray, where we would consider it to be
something that would not *greatly* benefit from decoupling its release
needs from that of Python (it's also true that it possibly introduces
complications for Python releases if they need to synch with array
schedules, but being inconsiderate louts, we don't care much about
that). And when one considers that the move to multicore and 64-bit
processors will introduce the need for significant changes in the
internals to take advantage of these capabilities, it is unlike we will
see a quiescent, maintenance-level state for an array package for some
time. In short, this issue is a distraction at the moment and will only
sap energy from what needs to be done to unify the array packages.
So what about supporting arrays as an interchange format? There are a
number of possibilities to consider, none of which require inclusion of
arrays into the core. It is possible for 3rd party extensions to
optionally support arrays as an interchange format through one of the
following mechanisms:
a) So long as the extension package has access to the necessary array
include files, it can build the extension to use the arrays as a format
without actually having the array package installed. The include files
alone could be included into the core (Guido has previously been
receptive to doing this though at this meeting he didn't seem quite as
receptive instead suggesting the next option) or could be packaged with
extension (we would prefer the former to reduce the possibilities of
many copies of include files). The extension could then be
successfully compiled without actually having the array package
present. The extension would, when requested to use arrays would see if
it could import the array package, if not, then all use of arrays would
result in exceptions. The advantage of this approach is that it does
not require that arrays be installed before the extension is built for
arrays to supported. It could be built, and then later the array
package could be installed and no rebuilding would be necessary.
b) One could modify the extension build process to see if the package
is installed and the include files are available, if so, it is built
with the support, otherwise not. The advantage of this approach is that
it doesn't require the include files be included with the core or be
bundled with the extension, thus avoiding any potential version
mismatches. The disadvantage is that later adding the array package
require the extension to be rebuilt, and it results in more complex
build process (more things to go wrong).
c) One could provide the support at the Python level by instead relying
on the use of buffer objects by the extension at the C level, thus
avoiding any dependence on the array C api. So long as the extension
has the ability to return buffer objects containing the putative array
data to the Python level and the necessary meta information (in this
case, the shape, type, and other info, e.g., byteswapping, necessary to
properly interpret the array) to Python, the extension can provide its
own functions or methods to convert these buffer objects into arrays
without copying of the data in the buffer object. The extension can try
to import the array package, and if it is present, provide arrays as a
data format using this scheme. In many respects this is the most
attractive approach. It has no dependencies on include files, build
order, etc. This approach led to the suggestion that Python develop a
buffer object that could contain meta information, and a way of
supporting community conventions (e.g., a name attribute indicating
which conventions was being used) to facilitate the interchange of any
sort of binary data, not just arrays. We also concluded that it would
be nice to be able create buffer objects from Python with malloced
memory (currently one can only create buffer objects from other objects
that already have memory allocated; there is no way of creating newly
allocated, writable memory from Python within a buffer object; one can
create a buffer object from a string, but it is not writable).
Nevertheless, if an extension is written in C, none of these changes
are necessary to make use of this mechanism for interchange purposes
now. This is the approach we recommend trying. The obvious case to
apply it to is PIL as test case. We should do this ourselves and offer
it as a patch to PIL. Other obvious cases are to support image
interchange for GUIs (e.g., wxPython) and OpenGL.
2) Scalar support, rank-0 and related. Travis and I agreed (we
certainly seek comments on this conclusion; we may have forgotten about
key arguments arguing for one the different approaches) that the
desirability of using rank-0 arrays as return values from single
element indexing depends on other factors, most importantly Python's
support for scalars in various aspects. This is a multifaceted issue
that will need to be determined by considering all the facets
simultaneously. The following tries to list the pro's and con's
previously discussed for returning scalars (two cases previously
discussed) or rank-0 arrays (input welcomed).
a) return only existing Python scalar types (cast upwards except for
long long and long double based types)
Pros:
- What users probably expect (except matlab users!)
- No performance hit in subsequent scalar expressions
- faster indexing performance (?)
Cons:
- Doesn't support array attributes, numeric behaviors
- What do you return for long long and long double? No matter what
is done, you will either lose precision or lose consistency. Or you
create a few new Python scalar types for the unrepresentable types?
But, with subclassing in C the effort to create a few scalar types is
very close to the effort to create many.
b) create new Python scalar types and return those (one for each basic
array type)
Pros:
- Exactly what numeric users expect in representation
- No peformance hit in subsequent scalar expressions
- faster indexing performance
- Scalars have the same methods and attributes as arrays
Cons:
- Might require great political energy to eventually get the
arraytype with all of its scalartype-children into the Python core.
This is really an unknown, though, since if the arrayobject is in the
standard module and not in the types module, then people may not care
(a new type is essentially a new-style class and there are many, many
classes in the Python standard library). A good scientific-packaging
solution that decreases the desireability of putting the arrayobject
into the core would help alleviate this problem as well.
- By itself it doesn't address different numeric behaviors for the
"still-present" Python scalars throughout Python.
c) return rank-0 array
Pros:
- supports all array behaviors, particularly with regard to numerical
processing, particularly with regard to ieee exception handling (a
matter of some controversy, some would like it also to be len()=1 and
support [0] index, which strictly speaking rank-0 arrays should not
support)
Cons:
- Performance hit on all scalar operations (e.g., if one then does
many loops over what appears to be a pure scalar expression, use of
rank-0 will be much slower than Python scalars since use of arrays
incurs significant overhead.
- Doesn't eliminate the fact that one can still run into different
numerical behavior involving operations between Python scalars.
- Still necessary to write code that must deal with Python scalars
"leaking" into code as inputs to functions.
- Can't currently be used to index sequences (so not completely
usable in place of scalars)
Out of this came two potential needs (The first isn't strictly
necessary if approach a is taken, but could help smooth use of all
integer types as indexes if approach b is taken):
If rank-0 arrays are returned, then Guido was very receptive to
supporting a special method, __index__ which would allow any Python
object to be used as an index to a sequence or mapping object. Calling
this would return a value that would be suitable as index if the object
was not itself suitable directly. Thus rank-0 arrays would have this
method called to convert its internal integer value into a Python
integer. There are some details about how this would work at the C
level that need to be worked out. This would allow rank-0 integer
arrays to be used as indices. To be useful, it would be necessary to
get this into the core as quickly as possible (if there are C API
issues that have lingering solutions that won't be solved right away,
then a greatly delayed implementation in Python would make this less
than useful).
We talked at some length about whether it was possible to change
Python's numeric behavior for scalars, namely support for configurable
handling of numeric exceptions in the way numarray does it (and
Numeric3 as well). In short, not much was resolved. Guido didn't much
like the stack approach to the exception handling mode. His argument (a
reasonable one) was that even if the stack allowed pushing and popping
modes, it was fragile for two reasons. If one called other functions in
other modules that were previously written without knowledge that the
mode could be changed, those functions presumed the previous behavior
and thus could be broken with mode change (though we suppose that just
puts the burden on the caller to guard all external calls with restores
to default behavior; even so, many won't do that leading to spurious
bug reports that may annoy maintainers to no end though no fault of
their own). He also felt that some termination conditions may cause
missed pops leading to incorrect modes. He suggested studying the use
of the decimal's use of context to see if it could used as a model.
Overall he seemed to think that setting mode on a module basis was a
better approach. Travis and I wondered about how that could be
implemented (it seems to imply that the exception handling needs to
know what module or namespace is being executed in order to determine
the mode.
So some more thought is needed regarding this. The difficulty of
proposing such changes and getting them accepted is likely to be
considerable. But Travis had a brilliant idea (some may see this as
evil but I think it has great merit). Nothing prevents a C extension
from hijacking the existing Python scalar objects behaviors. Once a
reference is obtained to an integer, float or complex value, one can
replace the table of operations on those objects with whatever code one
wishes. In this way an array package could (optionally) change the
behavior of Python scalars. In this way we could test the behavior of
proposed changes quite easily, distribute that behavior quite easily in
the community, and ultimately see if there are really any problems
without expending any political energy to get it accepted. Once seeing
if it really worked (without "forking" Python either), would place us
in a much stronger position to have the new behaviors incorporated into
the core. Even then, it may never prove necessary if can be so
customized by the array package. This holds out the potential of making
scalar/array behavior much more consistent. Doing this may allow option
a) as the ultimate solution, i.e., no changes needed to Python at all
(as such), no rank-0 arrays. This will be studied further. One possible
issue is that adding the necessary machinery to make numeric scalar
processing consistent with that of the array package may introduce
significant performance penalties (what is negligible overhead for
arrays may not be for scalars).
One last comment is that it is unlikely that any choice in this area
prevents the need for added helper functions to the array package to
assist in writing code that works well with scalars and arrays. There
are likely a number of such issues. A common approach is to wrap all
unknown objects with "asarray". This works reasonably well but doesn't
handle the following case: If you wish to write a function that will
accept arrays or scalars, in principal it would be nice to return
scalars if all that was supplied were scalars. So functions to help
determine what the output type should be based on the inputs would be
helpful, for example to distinguish from when someone provided a rank-0
array as an input (or rank-1 len-1 array) and an actual scalar if
asarray happens to map this to the same thing so that the return can
properly return a scalar if that is what was originally input. Other
such tools may help writing code that allows the main body to treat all
objects as arrays without needing checks for scalars.
Other miscellaneous comments.
The old use of where() may be deprecated and only "nonzero"
interpretation will be kept. A new function will be defined to replace
the old usage of where (we deem that regular expression search and
replaces should work pretty well to make changes in almost all old
code).
With the use of buffer objects, tostring methods are likely to be
deprecated.
Python PEPs needed
===================
From the discussions it was clear that at least two Python PEPs need to
be written and implemented, but that these needed to wait until the
unification of the arrayobject takes place.
PEP 1: Insertion of an __index__ special method and an as_index slot
(perhaps in the as_sequence methods) in the C-level typeobject into
Python.
PEP 2: Improvements on the buffer object and buffer builtin method so
that buffer objects can be Python-tracked wrappers around allocated
memory that extension packages can use and share. Two extensions are
considered so far. 1) The buffer objects have a meta attribute so that
meta information can be passed around in a unified manner and 2) The
buffer builtin should take an integer giving the size of writeable
buffer object to create.
