[Numpy-discussion] Array Metadata

[Scott Gilbert]

I got back late last night, and there were lots of things I wanted to
comment on.  I've put parts of several threads into this one message since
they're all dealing with the same general topic:


Perry Greenfield wrote:
>
> I'm not sure how the support for large data sets should be handled.
> I generally think that it will be very awkward to handle these
> until Python does as well. Speaking of which...
>

I agree that it's going to be difficult to have general support for large
PyBufferProcs objects until the Python core is made 64 bit clean.  But
specific support can be added for buffer types that are known in advance. 
For instance, the bytes object PEP proposes an alternate way to get a 64
bit length, and similar support could easily be added to Numarray.memory,
mmap.mmap, and whatever else on a case by case basis.  So you could get a
64 bit pointer from some types of buffers before the rest of Python becomes
64 bit clean.

If the ndarray consumer (wxWindows for instance) doesn't recognize the
particular implementation, it has to stick with the limitations of the
standard PyBufferProcs and assume a 32 bit length would suffice.


Travis Oliphant wrote:
>
> I prefer  __array_data__   (it's a common name for Numeric and
> numarray,  It can be interpreted as a sequence object if desired). 
>

So long as everyone agrees it doesn't matter what name it is.  Sounds like
__array_data__ works for everyone.


>
> I also like __array_typestr__  or __array_typechar__ better as a name.
>

A name is a name as far as I'm concerned.  The name __array_typestr__ works
for me.  The name __array_typechar__ implies a single character, and that
won't be true.


>
> Don't like 'D' for long double.  Complex floats is already
> using it.  I'm not sure I like the idea of moving to two
> character typecodes at this point because it indicates more
> internal changes to Numeric3 (otherwise we have two typecharacter
> standards which is not a good thing).  What is wrong with 'g'
> and 'G'  for long double and complex long double respectively. 
>

Nothing in this array protocol should *require* internal changes to either
Numeric3 or Numarray.  I suspect Numarray is going to keep it's type
hierarchy, and Numeric3 can use single character codes for it's
representation if it wants.  However, both Numeric3 and Numarray might
(probably would) have to translate their internal array type specifiers
into the agreed upon "type code string" when reporting out this attribute.

The important qualities __array_typestr__ should have are:

    1) Everyone should agree on the interpretation.  It needs to be
       documented somewhere.  Third party libraries should get the
       same __array_typestr__ from Numarray as they do from Numeric3.

    2) It should be sufficiently general in it's capabilities to
       describe a wide category of array types.  Simple things
       should be simple, and harder things should be possible.

       An ndarray of double should have a simple common well
       recognized value for __array_typestr__.  An ndarray of
       a multi-field structs should be representable too.

> 
> >
> >  __array_complicated__
> >
> 
> I don't see the utility here I guess,  If it can't be described by a 
> shape/strides combination then how can it participate in the protocol?
>

I'm not married to this one.  I don't know if Numarray or Numeric3 will
ever do such a thing, but I can imagine more complicated schemes of
arranging the data than offset/shape/strides are capable of representing. 
So this is forward compatibility with "Numarric4" :-).  Pretty
hypothetical, but imagine that typically Numarric4 can represent it's data
with offset/shape/strides, but for more advanced operations that falls
apart.  I could bore you with a detailed example...

The idea is that if array consumers like wxPython were aware that more
complicated implementations can occur in the future, they can gracefully
bow out and raise an exception instead of incorrectly interpreting the
data.  If you need it later, you can't easily add it after the fact.

Take it or leave it I guess - it's possibly a YAGNI.


> 
> After more thought,  I think here we need to also allow the
> "c-type" independent way of describing an array (i.e. numarray
> introduced 'c4' for a complex-valued 4 byte itemsize array).
> So, perhaps __array_ctypestr_  and __array_typestr__ should be
> two ways to get the information (or overload the __array_typestr__
> interface and reequire consumers to accept either style).
>

I don't understand what you are proposing here.  Why would you want to
represent the same information two different ways?



Perry Greenfield wrote:
>
> I think we need to think about what the typecharacter is supposed
> to represent. Is it the value as the user will see it or to indicate
> what the internal representation is? These are two different things.
>

I think __array_typestr__ should accurately represent the internal
representation.  It is not intended for typical end users.  The whole of
the __array_*metadata*__ stuff is intended for third party libraries like
wxPython or PIL to be able to grab a pointer to the data, calculate
offsets, and cast it to the appropriate type without writing lots of
special case code to handle the differences between Numeric, Numarray,
Numeric3, and whatever else.


>
> Then again, I'm not sure how this info is exposed to the user; if it
> is appropriately handled by intermediate code it may not matter. For
> example, if this corresponds to what the user will see for the type,
> I think it is bad. Most of the time they don't care what the internal
> representation is, they just want to know if it is Int16 or whatever;
> with the two combined, they have to test for both variants.
>

Typical users would call whatever attribute or method you prefer (.type()
or .typecode() for instance), and the type representation could be classes
or typecodes or whatever you think is best.

The __array_typestr__ attribute is not for typical users (unless they start
to care about the details under the hood).  It's for libraries that need to
know what's going on in a generic fashion.  You don't have to store this
attribute as separate data, it can be a property style attribute that
calculates it's value dynamically from your own internal representation.


Francesc Altet wrote:
> 
> Considering that heterogenous data is to be suported as well, and
> there is some tradition of assigning names to the different fields,
> I wonder if it would not be good to add something like:
> 
> __array_names__ (optional comma-separated names for record fields)
> 

I really like this idea.  Although I agree with David M. Cooke that it
should be a tuple of names.  Unless there is a use case I'm not
considering, it would be preferrable if the names were restricted to valid
Python identifiers.


Travis Oliphant wrote:
> 
> After more thought,  I think using the struct-like typecharacters
> is not a good idea for the array protocol.    I think that the
> character codes used by the numarray record array:  kind_character 
> + byte_width is better.  Commas can separate heterogeneous data.
> The problem is that if the data buffer originally came from a
> different machine or saved with a different compiler (e.g. a mmap'ed
> file), then the struct-like typecodes only tell you the c-type that
> machine thought the data was.  It does not tell you how to interpret
> the data on this machine. 
>

The struct module has a portable set of typecodes.  They call it
"standard", but it's the same thing.  The struct module let's you specify
either standard or native.  For instance, the typecode for "standard long"
("=l") is always 4 bytes while a "native long" ("@l") is likely to be 4 or
8 bytes depending on the platform.  The __array_typestr__ codes should
require the "standard" sizes.  There is a table at the bottom of the
documentation that goes into detail:

    http://docs.python.org/lib/module-struct.html

The only problem with the struct module is that it's missing a few types...
(long double, PyObject, unicode, bit).


> 
> I also think that rather than attach < or > to the start of the
> string it would be easier to have another protocol for endianness.
> Perhaps something like:
> 
> __array_endian__  (optional Python integer with the value 1 in it).
> If it is not 1, then a byteswap must be necessary. 
>

This has the problem you were just describing.  Specifying "byteswapped"
like this only tells you if the data was reversed on the machine it came
from.  It doesn't tell you what is correct for the current machine.

Assuming you represented little endian as 0 and big endian as 1, you could
always figure out whether to byteswap like this:

    byteswap = data_endian ^ host_endian

Do you want to have an __array_endian__ where 0 indicates "little endian",
1 indicates "big endian", and the default is whatever the current host
machine uses?  I think this would work for a lot of cases.

A limitation of this approach is that it can't adequately represent
struct/record arrays where some fields are big endian and others are little
endian.



> 
> Bool               -- "b%d" % sizeof(bool)
> Signed Integer     -- "i%d" % sizeof(<some int>)
> Unsigned Integer   -- "u%d" % sizeof(<some uint>)
> Float              -- "f%d" % sizeof(<some float>)
> Complex            -- "c%d" % sizeof(<some complex>)
> Object             -- "O%d" % sizeof(PyObject *)
>          --- this would only be useful on shared memory
> String             -- "S%d" % itemsize
> Unicode            -- "U%d" % itemsize
> Void               -- "V%d" % itemsize   
> 

The above is a nice start at reinventing the struct module typecodes.  If
you and Perry agree to it, that would be great.  A few additions though:

I think you're proposing that "struct" or "record" arrays would be a
concatenation of the above strings.  If so, you'll need an indicator for
padding bytes.  (You probably know this, but structs in C frequently have
wasted bytes inserted by the compiler to make sure data is aligned on the
machine addressable boundaries.)

I also assume that you intend the ("c%d" % itemsize) to always represent
complex floating point numbers.  That leaves my favorite example of complex
short integer data with no way to be represented...  I guess I could get by
with "i2i2".

How about not having a complex type explicitly, but representing complex
data as something like:

     __array_typestr__ = "f4f4
     __array_names__ = ("real", "imag")

Just a thought...  I do like it though.


I think that both Numarray and Numeric3 are planning on storing booleans in
a full byte.  A typecode for tightly packed bits wouldn't go unused
however...


> 
> 1) How do you support > 2Gb memory mapped arrays on 32 bit systems
> and other large-object arrays only a part of which are in memory at
> any given time
> 

Doing this well is a lot like implementing mmap in user space.  I think
this is a modification to the buffer protocol, not the array protocol.  It
would add a bit of complexity if you want to deal with it, but it is
doable.

Instead of just grabbing a pointer to the whole thing, you need to ask the
object to "page in" ranges of the data and give you a pointer that is only
valid in that range.  Then when you're done with the pointer, you need to
explicitly tell the object so that it can write back if necessary and
release the memory for other requests.  Do you think Numeric3 or Numarray
would support this?  I think it would be very cool functionality to have.

>
> (there is an equivalent problem for > 8 Eb (exabytes) on 64 bit
> systems, an Exabyte is 2^60 bytes or a giga-giga-byte).
> 

I think it will be at least 10-20 years before we could realisticly exceed
a 64 bit address space.  Probably a lot longer.  That's a billion times
more RAM than any machine I've ever worked on, and it's a million times
more bytes than any RAID set I've worked with.  Are there any super
computers approaching this level?  Even at Moore's law rates, I'm not
worried about that one just yet.

> 
> But,  I've been thinking about the array protocol and thinking that
> it would be a good thing if this became universal.  One of the ways
> to make it universal is by having something that follows it in the
> Python core.
> 
> So, what if we proposed for the Python core not something like
> Numeric3 (which would still exist in scipy.base and be everybody's
> favorite array :-) ), but a very minimal array object (scaled back
> even from Numeric) that followed the array protocol and had some
> C-API associated with it.
> 
> This minimal array object would support 5 basic types ('bool',
> 'integer', 'float', 'complex', 'Object').   (Maybe a void type
> could be defined and a void "scalar" introduced (which would be
> the bytes object)).  These types correspond to scalars already
> available in Python and so the whole 0-dim array Python scalar
> arguments could be ignored.   
>

I really like this idea.  It could easily be implemented in C or Python
script.  Since half it's purpose is for documentation, the Python script
implementation might make more sense.

Additionally, a module that understood the defaults and did the right thing
with the metadata attributes would be useful:

    def get_ndims(a):
        return len(a.__array_shape__)

    def get_offset(a):
        if hasattr(a, "__array_offset__"):
            return a.__array_offset__
        return 0

    def get_strides(a):
        if hasattr(a, "__array_strides__"):
            return a.array_strides
        # build the default strides from the shape 

    def is_c_contiguous(a):
        shape = a.__array_shape__
        strides = get_strides(a)
        # determine if the strides indicate it is contiguous

    def is_fortran_contiguous(a):
        # similar to is_c_contiguous

etc...

Thes functions could be useful for third party libraries to work with *any*
of the array packages.



> 
> An alternative would be to "add" multidimensionality to the array
> object already part of Python, fix it's reallocating with an exposed
> buffer problem, and add the array protocol.
> 

I'd recommend not breaking backward compatibility on the array.array
object, but adding the __array_*metadata*__ attributes wouldn't hurt
anything.  (The __array_shape__ would always be a tuple of length one, but
that's allowed...).



Magnus Lie Hetland wrote:
> 
> Wohoo! Niiice :)
> 
> (Okay, a bit "me too"-ish, but I just wanted to contribute some
> enthusiasm ;)
> 

I completely agree!  :-)



Cheers,
    -Scott